Installs only what it has checked. Removes only what it installed.
arospkg is a package manager for AROS, used from the Shell with the pkg command. It installs software from AROS Archives through a small public index, checks every download against that index, and removes packages without taking the files you changed.
The Shell client is the first front end, not the only intended one. Every decision about verifying, recording, refusing and recovering lives in one library, and pkg is a front end to it, so a graphical one would re-make none of them. The index already carries the categories, summaries and search that browsing needs.
Work in progress, to be released soon. Built from one source for two ABIs: ABIv1, which mainline AROS x86_64 uses, and ABIv11, which AROS One uses.
AROS Shell on AROS One, recorded pkg output. Nothing runs here.
What pkg is doing
-
Read the index entry
- id
- zaphod
- version
- 1.3
- arch
- x86_64
- size
- 218782
- sha256
- 603e292457c3439f32709380bdd1d9c72b06536d013eebc01541a0bf0fa75448
-
Connect to archives.arosworld.org
HTTPS only. The certificate has to match this host.
-
Follow the redirect
To arosarchives.os4depot.net, whose certificate is checked again against its own name.
-
Download into zaphod.zip.part
218782 bytes (an illustration: pkg shows no progress while it downloads)
-
Check the size, then the SHA-256
Both must match the index. Only then does zaphod.zip.part become the cached file.
-
Check every path in the archive
Before a byte is written. On AROS a colon anywhere makes a path absolute, and one bad path rejects the whole archive.
-
Stage, then move into place
The plan is written and read back, then the whole tree moves to SYS:Packages/zaphod in one step.
-
Record, then commit
50 files and their hashes go into SYS:Packages/db/installed/zaphod.json. The commit comes last.
pkg install printed, and the right side shows the steps behind those lines. If the archive is already in the cache, it isn't downloaded again (the cached copy is still checked), and pkg prints only installed zaphod. When the replay ends, type a pkg command at the prompt or pick one. Every response is output recorded from a real run. Nothing runs in your browser.An index, not a mirror.
AROS software lives in AROS Archives: 1,895 uploads with no dependency data and no install rules. arospkg doesn't copy any of it. It publishes a small index that points at each archive where it already lives and records the size and SHA-256 the download has to match.
The index is public, at github.com/tomaszstaniak/arospkg-index. It holds metadata and nothing else. Before a package goes in, its archive has been downloaded, hashed and opened, and the index only lists what the client can actually handle.
arospkg-index
Public, on GitHub. Metadata only.
zaphod 1.3 x86_64
size 218782
sha256 603e2924…fa75448
AROS Archives
archives.arosworld.org, served from arosarchives.os4depot.net. The file stays here.
zaphod-v1.3-.x86_64-aros-v11.zip
pkg, on AROS
Compares the download with the entry and keeps it only if the size and hash both match.
pkg update fetching https://raw.githubusercontent.com/tomaszstaniak/arospkg-index/main/index.json index updated: SYS:Packages/db/index.json pkg search zaphod zaphod 1.3 x86_64 A simple binary file editor pkg install zaphod fetching https://archives.arosworld.org/share/development/edit/zaphod-v1.3-.x86_64-aros-v11.zip redirect -> https://arosarchives.os4depot.net/download.php?file=development/edit/zaphod-v1.3-.x86_64-aros-v11.zip verified and cached SYS:Packages/cache/zaphod.zip installed zaphod pkg list zaphod pkg remove zaphod removed zaphod
A proposed package standard.
Proposal — not yet implemented or adopted
Authors describe how their software should be installed, so users and package maintainers don't have to rediscover it.
For the packages in the index, working out what they install and what they need meant downloading each archive, hashing it and opening it by hand. Of an ordinary four-package sample, half couldn't be approved at all: one was a developer SDK the manifest format can't describe, and one needed libraries that no package anywhere can supply, because they belong to the distribution rather than to any package. That second one was a gap in our own format, not in the archive, and closing it is where the check for what a machine already provides came from.
An ordinary ZIP or LHA, with a manifest inside
No new archive format. The file stays readable by hand, with UnArc, and by anyone who has never heard of pkg.
ABI, architecture, dependencies and install rules, stated outright
The ABI carries weight rather than decoration: pkg already refuses a package built for the other one before downloading anything. The install rules describe what should end up where, rather than how to get there.
Program files, configuration and user data told apart
Today the manager hashes files and warns, which is safe but blunt. It can't tell a config you edited from a corrupted binary, so it keeps both and explains neither. An author writing kind = "config" once turns that warning into the right behaviour.
Where the source came from
The repository and revision it was built from, or an archive and its hash, plus which SDK built it and that SDK's build date. Rebuilding then becomes a task rather than an excavation: whoever has the toolchain rebuilds, and everyone else gets the fresh binary through pkg update. Software for AROS is cross-built from a host machine.
These fields record what a binary was made with, which is the question worth answering when something breaks. They are not an expiry date and not a compatibility test: one built a year ago may work perfectly, and one built last week may not. The two builds need this differently, too. ABIv11 promises backwards compatibility, so rebuilding is occasional, while mainline promises none, which makes a shelf of prebuilt binaries there a perishable thing.
It works alongside the index
Packages that exist today keep working through the overlay, which needs nothing from their authors. The standard is for what comes next.
No install scripts in the first version
A manifest describes the state to reach; it doesn't run code. Every promise the manager makes about recovering from an interruption depends on knowing the whole set of changes in advance and writing it into the plan. An arbitrary script makes that plan incomplete.
What a manifest would look like
exampleapp below is invented, so that no real program has dependencies or sources attributed to it that it doesn't have. The syntax is proposed. pkg doesn't read this file today.
# exampleapp.toml — an INVENTED package, shown to give the shape. # Proposed syntax. pkg does not read this file today. manifest = 1 # the standard's version, not the program's [package] id = "exampleapp" # stable, lowercase, never carries a version name = "ExampleApp" # what a person calls it summary = "An invented program, used here to show the manifest" category = "utility/misc" version = "2.4.1" # upstream's version revision = 3 # OUR packaging; bumped when upstream did not change license = "GPL-2.0-or-later" [build] arch = "x86_64" abi = "v11" # v1 = mainline, v11 = current distributions # Other packages. The index must contain them or this package is not published. depends = ["examplelib"] # What the MACHINE must already provide. A different question with a different # answer: no package can supply these. [[requires_system]] type = "library" id = "SDL.library" [[requires_system]] type = "library" id = "crt.library" min_version = 5 [install] dir = "SYS:Packages/exampleapp" # everything the package owns lives here icon = "ExampleApp.info" # the drawer icon, which sits beside the drawer # Told apart so removal and upgrade know what they may touch. [[files]] path = "ExampleApp" kind = "program" # replaced on upgrade [[files]] path = "ExampleApp.prefs" kind = "config" # written once; yours afterwards [[files]] path = "saves/" kind = "userdata" # the package never owns what appears here [source] repo = "https://example.invalid/exampleapp.git" revision = "9f2c1ab4e7d05c38b1f6a0d9e4c7b2a8f1d3e6c0" # or, for a tarball rather than a repository: # archive = "https://example.invalid/exampleapp-2.4.1.tar.gz" # archive_sha256 = "…" sdk = "AROS One 1.3 SDK" sdk_date = "2018-04-28" toolchain = "x86_64-aros-gcc 10.5.0"
The packaging revision is separate from the version
Upstream's version answers which program this is. The packaging revision answers which attempt at packaging it. Fixing a wrong icon path bumps the second and not the first, and without the split there's no way to ship that fix.
What another package provides, and what the machine provides
These are different questions, not two spellings of one. "No package in the index provides this" can be fixed by publishing a package. "This machine doesn't have it" can't. Conflating them means telling someone to wait for a package that will never exist. The machine half is the part that works today.
A minimum version, where one is known
Present isn't the same as present and new enough. Lopan's real manifest sets no minimum anywhere, deliberately: the versions it was built against aren't recorded in the archive, and a guess would refuse installs that work.
The icon sits beside the drawer
Not inside it. An AROS quirk that cost this project a manifest revision. It also matters more than it sounds: started from a Shell somewhere else, Lopan looks for its tiles in data/ relative to wherever you are and prints Failed to load a tile set, code 1, so you have to change directory first. Launched from an icon, Workbench points it at its own drawer and it just works. The icon is the fix, and it isn't built yet.
Three things, easily blurred
- Manifest
- Describes the package, and travels inside it.
- Index
- Points at an archive and records its hash.
- Local registry
- Describes the actual installation on one machine, including which requirements were satisfied and which were left undetermined.
Where the ideas come from
Neither of these is a claim of compatibility with anything.
Homebrew keeps package definitions as readable files in a Git repository, identifies sources by URL and checksum, and has a test block that checks the installed thing actually works rather than that files landed. What that gives us: definitions in Git as the source of truth with the index derived from them, checksums that are never optional, and the habit of treating "it installs" and "it runs" as two separate claims.
RPM is the package format, and the low-level tool and database that install one. DNF is the layer above: it resolves dependencies, manages repositories and decides what to fetch. What that gives us: the explicit vocabulary of requires, provides and conflicts, and the split itself, which is why every decision lives in one library and pkg is only a front end to it. One honest difference: in RPM's vocabulary, what we call a system requirement is a requirement nothing provides, because on AROS the distribution is the provider and no package expresses it. That is why the two had to be separate fields rather than one list with a flag.
Not copied: install scripts and triggers, for the recovery reason above; weak dependencies; epochs; and several versions of one package installed at once.
Independent repositories
A shared manifest would also make independent repositories possible. If the description travels with the package, an author, a community or a distribution could publish a metadata repository of its own — plain static files, no application server — and a client could read more than one, choosing the entries that match its ABI and architecture. This project's public index is already separate from where the binaries live, so this extends a split that exists rather than replacing anything.
None of it is built, and the limits matter more than the idea. A SHA-256 in an index proves a download matches that index. It doesn't authenticate whoever wrote the index, and no signature makes a program good or safe. Identifier collisions between repositories, which source wins, and stopping a stale index from being served in place of a current one are problems to design, not details to fill in.
The three layers stay separate: whoever hosts the binaries, the metadata repository pointing at them, and the client reading one or more of those. Binaries keep coming from AROS Archives. This project doesn't mirror or host them, and that isn't changing.
Nothing in this section exists in the client today, and none of it has been adopted by anyone. The proposal is kept with the project's other design notes.
Commands.
Every command in pkg 0.2, in the order its help screen lists them. All of them run from the AROS Shell, against the default root SYS:Packages.
- pkg update
- Fetches the index from arospkg-index over HTTPS and saves it as
SYS:Packages/db/index.json. If the download fails, or what arrives isn't a valid index, the copy already on disk is kept. - pkg search [term]
- Looks for the term in each package's id, summary and category, ignoring case, and prints each match with its version and architecture. The category is searched but not printed, so
pkg search gamefinds SDLPoP. With no term it lists the whole index. With no match it printsnothing matches. Packages built for the other ABI are left out, with a count of how many, and--all-abilists them marked[other ABI]as a diagnostic. - pkg list
- Prints the name of each installed package (one registry file each, in
SYS:Packages/db/installed/), ornothing installed. - pkg info id
- Prints the registry entry for one installed package as JSON: name, version, where it came from, the archive's hash, and one line for every file it installed, like this one from zaphod's:
{ "path":"AUTHORS", "sha256":"02672617a8ac7f1f464ff2eef9b57eca9a0c5e64bcaf756385a8581191f49e55", "size":1645 }sdlpop's record, covering 1074 files, is 139,646 bytes. - pkg install id
- Downloads the archive, or uses the copy already in the cache, and checks it against the index. Then it checks every path inside it, moves the package into
SYS:Packages/idin one step, and records each file with its hash. An id that isn't in the index is refused, because the index lists only reviewed packages. An installed package has to be removed before it can be installed again. A package built for the other ABI is refused before anything is downloaded, and an unknown or missing ABI is refused rather than guessed at. So is a package whose system requirements this machine doesn't meet. A requirementpkgcan't decide either way is installed with a warning, and the doubt is recorded alongside the package. - pkg remove id
- Hashes the package's files on disk, deletes the unchanged ones and removes directories left empty. Files you edited, and files it doesn't recognise, are kept with a warning and listed in a report. The report is written before the registry entry is deleted, so anything left behind can still be traced to its package.
- pkg unlock
- Clears a lock left behind by a run that died.
- pkg doctor [--retry]
- Says what is unresolved and why: interrupted transactions, and reports left by earlier removals. It never repairs anything itself.
--retryre-runs the ordinary recovery. There is deliberately no force option.
The only options for users are --help, which prints the list of commands, and --version, which prints the build, compiler, toolchain and SDK. The help screen also lists assert-hash, which is a test utility. The remaining options are for the test harness or for diagnosis: --all-abi lists packages for the other ABI, and --abi installs one anyway, which the help itself describes as not for use, since it won't start.
It never deletes a file it didn't install.
Removing a package
Every installed file is recorded with its hash. On removal, pkg hashes the files on disk first. Unchanged files are removed. Files you edited, and files it doesn't recognise, stay where they are with a warning, and a report lists what was left. In the test with one edited file: 49 removed, 1 kept.
When something is interrupted
Every install and removal is journalled. The next run repairs an interrupted one, working from the plan and from what is actually on disk. This was tested at every interruption point the code defines, including interrupting the recovery itself and a write failure at the moment the commit is recorded.
A file nobody registered
In one test, a file arospkg hadn't installed was sitting in a package's directory when an install was interrupted. The file survived. Rather than tidy up around it, pkg stopped and left the decision to a person.
A package for the other ABI
pkg knows which ABI it was built for, and that is the system's ABI as well, because a binary for the other one wouldn't have started. A package built for the other ABI is refused before anything is downloaded, and an entry whose ABI it doesn't recognise is refused rather than guessed at. An override exists, and it warns instead of pretending the result will be fine.
What the machine has to provide
Some things a package needs belong to the system, not to another package: a library the distribution ships. pkg checks the machine before downloading anything. If what's needed isn't there, it refuses and says the lack is the machine's. If it can't tell either way, it installs, warns, and records the doubt with the package, so pkg info still shows it later. Rounding an unanswered question up to "fine" would be the lie; refusing every time it can't be sure would make the manager useless. When everything is there, it says nothing at all: only a refusal or a doubt speaks.
When it can't finish
pkg doctor says what is unresolved and why, and pkg doctor --retry tries again. There is deliberately no command that force-deletes everything.
recovery: 1 transaction(s) to resolve
6aa05ed0-zaphod: install without commit -- rolling back
50 file(s) removed, 0 left because changed
recovered 1 interrupted transaction(s)
pkg: this system does not provide what the package needs
req-test-missing
arospkg-test-absent.library (not on this machine)
WARNING: 1 of this package's 1 system requirement(s) could not be
checked. It will be installed, and this is recorded with it.
datatype png: requirement type "datatype" is not one this client can check
installed req-test-undetermined
pkg can't decide. It says so, installs, and writes the doubt into the record rather than quietly calling it fine.pkg: wrong ABI for this system
zaphod
It is built for ABIv11; this system is ABIv1. It would install correctly and then fail to start. Use the ABIv11 build of pkg, or a package for ABIv1.
pkg install zaphod on mainline, where both packages in the index are for the other ABI. Nothing is downloaded.WARNING: --abi v11 overrides the check. This binary is v1, and a
package for another ABI installs correctly and then does
not start.
SYS:PkgTest/zaphod still exists -- left in place, transaction kept
pkg: recovery stopped short
SYS:PkgTest
1 transaction(s) could not be resolved. Nothing was cleaned up. Inspect db/transactions and run pkg doctor.
Where it stands.
arospkg is a command-line tool with no public release yet, and the first of two intended front ends onto the same library. What works has been run. What comes next is being built, and what comes after that is still a plan.
It is developed on mainline and built for ABIv1 and ABIv11. On AROS One, using the ABIv11 build, the full run is confirmed: fetching the index, search, download and installation of ZapHod, and launching its interface. That doesn't confirm the rest of the catalogue, or the same packages on mainline.
pkg from the index on AROS One and then opened: ZapHod, a hex editor, and Lopan, a Mahjong clone. Lopan was started once, to a drawn board and a working close gadget, which isn't a claim that the game is fully playable. Screenshots illustrate. Each run is evidenced by a report carrying its run id and the binary's own SHA-256, because a picture can't say whose output it is.Works today
Tested on mainline AROS x86_64, and on AROS One with the ABIv11 build, both in QEMU.
pkg update,search,install,listandremove.- Two builds from one source: ABIv1 for mainline, and a build for the ABI AROS One uses.
- On AROS One, the whole chain on a persistent disk, ending with the installed editor open.
- A check of what the machine already provides, before anything is downloaded. A package whose requirements aren't met is refused, with the lack named as the machine's. A requirement
pkgcan't decide is installed with a warning, and the doubt is written down with the package, so it is still visible later. - An ABI gate: a package built for the other ABI is refused before anything is downloaded, and an unknown or missing ABI is refused rather than guessed at. Search leaves other-ABI packages out and says how many.
- The same refusals measured on the older OpenSSL: a certificate for a host the server doesn't have, a plain
http://address refused before a socket opens, and no download at all without the CA bundle. - HTTPS only, with a required CA bundle. Certificates must match the host, and every redirect is checked again.
- Size and SHA-256 checked against the index before anything is unpacked.
- A corrupted archive in the cache is refused. It isn't repaired or quietly downloaded again.
- Its own ZIP reader, which checks every path before writing.
- A record of every installed file and its hash.
- Removal that keeps edited and unknown files.
- A journal, and recovery after an interruption. On AROS One,
doctor --retryrolled back 1074 files with none left behind and the other package untouched. pkg doctorandpkg doctor --retry.
Being built
Designed, not finished. No dates.
- Installing files into
LIBS:andFonts:.
Where it's going
Intent. None of this exists yet.
- Upgrades, and Workbench drawer icons for installed packages. Both are designed.
- Resolving what one package needs from another. It is deferred rather than missing: of 32 ABIv11 packages checked against AROS One's 84 libraries, one reference went unresolved, and it was
bsdsocket.library, which AROSTCP provides. That is what one method found in that sample, and a scan of this kind can't see a name a program assembles as it runs, so it is a reason not to build a resolver yet rather than a finding about the catalogue. - A graphical front end built on the same library as
pkg, so it doesn't re-make any of its decisions. - Something closer to a storefront: browsing by category, searching descriptions, a row of recently added software. The catalogue already has 12 top-level categories, 50 sub-categories and a description for each of its 168 x86_64 and aarch64 candidates.
- Perhaps a hand-picked selection, labelled as hand-picked.
- Community signals such as likes, much later if at all. They would never count as a sign that a package can be trusted, and installing would never depend on them.
Along the way, two fixes went into AROS mainline: a library that was never being built, and a 64-bit sign-extension bug that broke every TLS connection through the generic socket path.
Snapshot: . arospkg is an independent project by Tomasz Staniak. It is not affiliated with AROS, AROS Archives or any AROS distribution.