- Scale700+ open European opportunities
- AcceptanceSix deterministic publication rules
- StateSQLite lifecycle with provenance
- RecoveryRestore-tested, checksummed snapshots
- Quality89.9% critical-path coverage
The Problem I Kept Repeating
Stack: Python · TypeScript · Next.js · SQLite · Docker · GitHub Actions
Searching for early-career technology roles across Europe meant checking the same facts over and over. Results mixed internships with senior positions, combined different hiring cycles, included unrelated work, and often pointed outside Europe.
I wanted a directory focused on 2027 internships and New Grad roles, but I did not want to maintain another list by hand. European Tech Opportunities 2027 became both a public website and the data pipeline behind it.
The collection itself was only the starting point. A useful directory also needs to answer harder questions: Is there enough evidence to publish this listing? Is it the same role found through another search? Does disappearance mean closure? Can the database be recovered after a bad update?
The project favors precision over maximum coverage. When the available evidence is ambiguous, the listing stays out rather than being guessed into the dataset.
What Qualifies for Publication
Every new candidate passes six checks:
- Employment type: the title explicitly identifies an internship or New Grad role.
- Posting date: the resolved date is May 1, 2026 or later.
- Seniority: senior, lead, principal, management, and similar configured terms cause rejection.
- Hiring cycle: explicit 2027 evidence is accepted and a conflicting year is rejected.
- Technology category: the title, or a narrow description fallback, matches a configured category.
- European location: the normalized location resolves explicitly to Europe or a supported European country.
Yearless listings need a clear policy. They may qualify when posted on or after May 1, 2026 and all other checks pass. That cutoff marks the project's current hiring window; it does not claim that a yearless role is labeled 2027. Graduation-year language alone is not accepted as internship-cycle evidence, and an explicit conflicting year always wins over recency.
The classifier returns reproducible decisions, not confidence scores. That reduces coverage, but it lets me explain why a role appears and test rule changes against known behavior.
From Search Configuration to Public Data
validated YAML searches
-> bounded guest-page collection
-> title and company prefilters
-> selected detail pages
-> normalization and classification
-> transactional SQLite state
-> website + README preview
The Python runner fetches searches asynchronously with limits on pacing, concurrency, retries, pages, response sizes, and result counts. Results begin as untrusted candidates; only normalized records that pass every publication rule can reach the database.
Collection is disabled by default behind an operator authorization interlock. That interlock records a decision; it does not grant permission to access the source. The project uses no credentials, sessions, browser automation, private endpoints, proxies, CAPTCHA bypasses, or anti-bot evasion, and an upstream challenge is a stop condition.
SQLite stores the accepted lifecycle state. The website and generated README only read from it: neither can discover, classify, close, reopen, or otherwise modify a listing. Search configuration determines where the pipeline looks; it does not weaken the evidence required for publication.
Identity, Provenance, and One Writer
A numeric LinkedIn job ID identifies each listing. Separate provenance rows record every configured search that discovered it, allowing overlapping searches to support one public job without creating duplicates.
Searches fetch concurrently, but the repository serializes persistence. Each search produces an isolated outcome, so one failure can record diagnostics without changing another search's accepted jobs or absence evidence.
A successful transaction records the run, upserts accepted listings, refreshes provenance, advances observation timestamps, and applies explicit unavailability evidence. Jobs remain open while at least one active search association supports them.
The Next.js application opens short-lived read-only SQLite connections. The README is intentionally incomplete—it contains the ten latest roles for each employment type—so it cannot be mistaken for a backup or reconstructed into the full database.
Disappearance Is Not Closure
Removing expired jobs looked simple until I considered what a missing search result actually proves. Ranking changes, pagination, query edits, and temporary source failures can all hide a listing that remains open.
Normal collection therefore rechecks missing known jobs directly. Only repeated 404 or 410 responses deactivate a search association, and the job closes only after every supporting association becomes inactive.
A daily auditor examines every stored job independently of its discovery searches. It checks the canonical public listing and, when needed, the guest detail endpoint:
valid listing -> keep or reopen
explicit 404 / 410 -> delete listing and provenance
"No longer accepting" notice -> delete listing and provenance
ambiguous response or failure -> preserve current state
These paths use different evidence. Per-search confirmation protects one provenance relationship against a transient detail failure; the auditor changes the job itself only after a direct listing-level closure signal. Authentication, rate-limit, server, parser, and transport failures are inconclusive and leave the known state untouched.
This conservative behavior is intentional. An uncertain external response should create a diagnostic, not a destructive transition disguised as certainty.
Operating the Pipeline
Normal CI is offline with respect to the external source. Independent workflows validate the Python application, Next.js site, and Docker images using fixtures and synthetic SQLite state.
The authorized schedule audits existing jobs before collecting candidates. Each accepted update produces a versioned SQLite snapshot with a SHA-256 checksum and schema metadata. The workflow downloads that exact artifact into a clean directory and proves it can be restored before advancing the latest-state pointer.
Generated README changes arrive as a review pull request rather than deploying immediately. After review, a manual workflow restores the approved snapshot, verifies its checksum on the VPS, acquires a deployment lock, keeps the previous database, and atomically replaces the canonical file. Short-lived website connections pick up the new state without an application rebuild.
This process is intentionally cautious because the database contains history that cannot be recreated from the public README.
Current Result
As of August 11, 2026, the directory contains 731 open opportunities:
- 340 internships;
- 391 New Grad positions.
The offline critical-path suite reports 89.9% combined statement and branch coverage, including 81.8% branch coverage and 95.0% classifier branch coverage.
Beyond the counts, the project ships cross-search deduplication, provenance, first- and last-seen history, conservative closure handling, a searchable and filterable Next.js directory, a bounded README preview, restore-tested snapshots, and locked atomic deployment.
The listing count shows that this is no longer a dataset I can inspect manually after every run. The coverage numbers matter for a different reason: classification and lifecycle changes can be checked against established behavior before they alter public state.
Lessons and Next Work
I began by thinking about search and filtering. Running the directory over time shifted my attention to identity, evidence, transitions, and recovery. Returning records is easy compared with deciding what they mean on the next run.
Deterministic rules suit this project because each accepted role can be traced back to the evidence and state changes that put it online. They are less flexible than probabilistic classification, but easier to test, review, and correct without losing that explanation.
The largest remaining infrastructure gap is off-provider recovery. Restricted VPS snapshots, GitHub artifacts, cache, and the previous deployment copy cover several mistakes, but the durable snapshots still share a provider with production. Replicating them to independent storage would protect against complete VPS, disk, or provider loss.
Additional job sources should arrive through explicit adapters while preserving the existing classification and lifecycle contracts. A new provider should expand discovery, not introduce a second definition of an acceptable role or another uncontrolled writer.