Data Lifeboat | Preserve Flickr Photos

Changelog

Added a temporary status page to the lifeboat, so we have a sensible "resting place" to land and show a creator what the next step in the process is.

Added inline photo count to the create lifeboat page, so you can see how many photos are in the lifeboat before you submit it.
Added a tier warning to the lifeboat page, so you can see when you've exceeded the photo count limit in the Alpha stage.

A whole bunch of mild backend and frontend things that added up to a nice big chunk of work, but small enough that they didn't get their own changelog entry. Pretty sure they were mostly good chnages though.

The site now has a clean new look (thanks Juwon)!!!
Added a survey link to the download page, so we can get feedback from people who download the Data Lifeboat.
A bunch of other cool UI updates (thanks George!)
Changed how we display permissions and safety information, so it's easier to see the breakdown.
A near infinite number of backend improvements (thanks Dan) since the last update.

Added a "Data Lifeboats You're Aboard" section to the homepage, so you can see what Data Lifeboats you're a part of. Then a preview were you can view the Data Lifeboat, and give consent to which photos are included (by general groups, no individual photo selection yet).

Enhanced URL validation and error handling: the system now gracefully processes invalid Flickr URLs (private albums, deleted content, etc.) without hanging, some error feedback to users. Would be nice to have this before you start the Data Lifeboat, but for the moment at least things don't hang! 😅
Switched the MAX PHOTOS limit from a hardcoded value to one we can adjust, and added a "tier" system to the site, so we can limit the number of photos that can be collected for each lifeboat, while we figure out system resources, scalability, and cost, etc.

Added priority and round robin queue selection to the queue. Tasks are plucked off the queue based on a combination of priority and round robin selection. "Quicker" tasks that allow people to start working with the Data Lifeboat (such as gathering the initial metadata) are given a higher priority than crunching through the slower tasks (such as building the zip package). The round robin part means we switch between Lifeboats in a round robin fashion, so we don't always pick the same Lifeboat to work on. There is also anti Lifeboat "starvation" by making sure low priority tasks still get processed (by making sure that for every X high priority tasks, we also process Y low priority tasks).

Stop the page reload/polling stalling when the zip package is being built, now we should get a "building zip" status and polling for when the zip is ready. 🤞

Made the gathering shallow and deep metadata progress indicators more accurate, and updating "inline" (with polling, fancy-pants websocket updating coming later). Also updating the progress bar without reloading the whole page.

For the moment only "staff" users can create lifeboats (those are users who are admins of the Data Lifeboat Flickr group), normal members of the group can log in, but not create lifeboats.
Added an "interstitial landing page" for when people hit a page that they need to be logged in to view. The page tells them we're going to take them to Flickr to authenticate, and then they'll be redirected back to the page they were trying to view. Rather than just immediately bump them there. This is needed so when someone comes in to give consent they a) know what's happening and b) we can work out where to drop them back if they can't view the Data Lifeboat.

Changed the consent page to show a more detailed breakdown of who needs to be contacted, and who we can comment on. And who we can comment on in split between people who have commenting turned off, and people who aren't members of the Datalifeboat Flickr group. As we'll only leave comments on photos of people who are members of the group and have commenting turned on.
For staff added a quick "Execution time" indicator to the bottom right of the page so we can see how long it takes the backend to process a request.
Adjusted the user login a bit so we persist the user's details a little bit longer.
Switching from an "Allow list" of IDs to access the site, to being a member of the Datalifeboat Flickr group.

Implemented a README v2. It used to be that the README file was "locked in" when the consent stage was hit, mainly so people know whatever they were consenting to wouldn't change. But now the creator can keep editing a second version of the README file as the consent comes in. The final README in the zip package should include both versions, but for the moment it only includes a couple of questions from the second version.
Added a migration script system, so when we add new fields to the database, we can migrate the existing data to the new format, and not have to throw the current data away.
Cleaned up url validation code for adding URLs to lifeboats, can add "naked" URLs and it'll add the https:// prefix.
Fix an inbetween state for the "consent progress" button in the contextual ui, when we've downloaded all the metadata, but not yet built the zip package.
Format the Privacy & Terms page a little nicer, there's still nothing there yet.
Add a Feedback page with minimal content.
Add a changelog page 😁