# Description of Changes
Combine the Policies and Pipelines pages into one, so we have the new
concept of Policies as Pipelines that always run which the user cannot
disable. What used to be Policies are now referred to as Templates, and
they allow you to create a new Pipeline more easily with the simple UI.
There's followup work to be done here to improve the template UIs
because they've not been touched in a long time, but I've considered
that beyond the scope of this merge. The only real changes I've made to
them in this PR is that they have a toggle for whether they're policies,
they now have a "Customise" button to kick you into the full Pipeline
editor, and I've removed the source selection. Previously, they
supported selecting as many sources as you liked, but that feature never
worked and is incompatible with the backend as it stands now, which only
allows for one source. Because of that, I've made it so that they can
only run in editor unless you open them in the custom pipeline editor,
where you can switch out which source it will use.
There's also another bit of followup to rename and remove all the
previous Policies code. Now that they've been combined into one, we
don't need a lot of the Policies code anymore, but also there's about
300 files in the frontend referencing policies in text/comments which
need to be updated to say pipelines. This is way more work than is
reasonable to do in this PR so I'll just do it in a new PR.
## Limitations
This PR is about the merging of the old Policies and Pipelines and I'm
considering enforcing the new definition of a Policy where it's only
modifiable by admins beyond the scope of this PR.
<img width="756" height="395" alt="image"
src="https://github.com/user-attachments/assets/d31be5ce-f1c9-46b3-8e8d-866e63f89a81"
/>
<img width="1507" height="793" alt="image"
src="https://github.com/user-attachments/assets/9ba8875f-8be5-4881-91cf-40e0bc1076dc"
/>
<img width="1508" height="787" alt="image"
src="https://github.com/user-attachments/assets/3e1da77b-a0c0-4262-aad3-16650098db81"
/>
---------
Co-authored-by: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com>
# Description of Changes
When #6697 merged, the CI didn't run for some reason so it was never
caught that the tool models were out of date. This PR updates them to
the correct state.
# Description of Changes
Fixes various bugs that affected SaaS (and some self-hosted):
- Refreshing on Editor caused the user to be redirected to Processor
- User was unable to access Processor in SaaS
- Deep link hijacking fixes
- Fix double prefix `/app/app` issue
- Fix going from tool -> editor -> processor -> editor putting you back
into tool
---------
Co-authored-by: Anthony Stirling <77850077+Frooodle@users.noreply.github.com>
# Description of Changes
Originally, I wanted to re-enable typed linting on our repo but using
Oxlint this time to avoid the memory and speed issues that ESLint was
causing. Unfortunately, it's not stable enough yet to actually use on
our repo (although it is close, I suspect it'll be stable enough fairly
soon). I was able to remove many of the unnecessary casts that it found
though, so even though this won't be enforced, it's still worth cleaning
up what I've found.
# Description of Changes
e2e Playwright tests are currently failing intermittently on all
platforms for different reasons, most notably WebKit, which seems to
fail much more often than the others. This PR attempts to fix the
issues. I've ran the e2e tests a few times now and they don't seem to be
inconsistent any more, but it's difficult to tell if all the issues are
genuinely fixed due to the inconsistent nature. As far as I can tell,
I've not broken anything though.
# Description of Changes
The Documents tab in the Processor is supposed to be available to all
Processor users, but because the API is built on top of the Audit data,
which is only for enterprise users, the API call always fails with 403.
This means that it never fills the query cache, so every time you go
back to the tab it has to reload all the data for a couple of seconds
(and will fail again). This fixes the API so that it's available to any
Processor user instead of just enterprise users. Also, the documents
data was only being written to the log on an enterprise license, so I've
changed it so that data is always tracked in the audit log because
otherwise the Documents tab would still be useless to non-enterprise
users.
The Audit Log tab was also available to all Processor users, but would
have the same issue where the table would never load because the API
would 403 as well. I've just made the Audit Log tab disabled for
non-enterprise users now. We might want to do something to signpost it a
bit more that it's an enterprise-specific feature, but it's better than
nothing for now.
# Description of Changes
Prettier takes about 10 seconds to run over our frontend folder, but
[Oxfmt](https://oxc.rs/docs/guide/usage/formatter.html) does an (almost)
identical job in 0.2 seconds. This PR converts our Prettier integration
to an equivalent Oxfmt integration. There's exactly 2 files in the
frontend folder that Oxfmt formats differently to Prettier so it'll
barely cause any disruption to the source.
I've removed the `--check` option from the frontend tool models
generator as part of this because we can do the same thing with Task
easily enough and Oxfmt isn't directly importable like Prettier since
it's a Rust binary instead of a JS library. Originally I was shelling
out to Oxfmt on single-file mode to keep it all in memory but it just
seemed more likely that there'd be config mismatches between that script
and the task so I think it's better this way.
# Description of Changes
Follow-on from #7334. Fix more `any` type usages and ban them in the
linter. We're starting to get down to only difficult folders left now,
so some of these fixes replace an excluded folder with a couple of
individual files to reduce scope to manageable levels.
There are two real behaviour changes in this PR because of bugs that
were never caught due to the lack of proper typing:
- In the Google Drive service, `lastModified` was always `undefined`
because it should have been read via `lastModifiedUtc`, which it now is.
This means that files being read from Google Drive should now accurately
retain their last modified date from Drive.
- In the error toasts, there was translation logic to try and make
friendlier error messages, but it'd never actually fire since it relied
on `i18n` being written to `globalThis`, which it never was. It now
imports the singleton instead so that translation should start working.
I also had to tweak the way that FitText works because it was relying on
`any` typing to mix refs between different places where they weren't
technically compatible but I've changed it to go via a function and the
behaviour doesn't change.
# Description of Changes
Currently in the Processor's Pipelines page, none of the tools which
require supporting files are usable because it's never been hooked up to
the new API to upload supporting files. This PR hooks it up to that so
all tools using supporting files work in the processor. I had to tweak
the type generation a little for this so we have a static map of which
params are for supporting files so we know to handle them differently.
The `Test with a file` button has to work a little differently than the
main run since it's running an ad-hoc pipeline so the files haven't
necessarily been saved to the server yet. In this case, it'll use
whatever local changes the user has made for those pipeline steps, and
for all other steps, it'll just use what's saved in the server.
# Description of Changes
Overlay PDFs and Change Metadata both crashed in the Processor because
they required `FilesModalContext` and `ViewerContext` respectively.
Neither of those contexts make sense to provide in the Processor because
there are no files in context and there is no Viewer, so redesign both
tool settings to only optionally require these contexts. Their behaviour
is unchanged in the Editor but they now work in the Processor (just
without the extra info about the active files, since there are none).
Also hooks up the Reorganise Pages settings so that it can be used from
Automate. The component already existed but just wasn't being used,
which just looks like an oversight.
# Description of Changes
This fixes the a11y violations that are currently failing in the
nightlies in dark mode. Now that we're down to 0 baseline, we can
require the a11y tests to pass in PRs before they merge, so I've changed
that, and I've also made it so that the nightly will report failures in
both light and dark mode instead of just light mode if that fails.
<img width="2560" height="838" alt="image"
src="https://github.com/user-attachments/assets/b0322182-f6ff-4dea-9a1c-5a6c9c9c5439"
/>
# Description of Changes
Frontend CI is currently set to run `task frontend:check:all`, which
runs the tests, and then manually runs them a second time with coverage
enabled. This adds about 4 minutes onto the runtime of the frontend
tests for no reason. This PR changes the `frontend:test` rule to respond
to the `COVERAGE` and `CI` env vars to enable coverage analysis of the
frontend tests if the setting tells them to.
# Description of Changes
Currently when calculating the output file type for some tools, the
system will get it wrong because it doesn't know about what the default
parameters in tools are, so if it doesn't have a value for some key,
it'll just bail out and say "it might not be compatible". This PR adds
logic to `ToolIO` to read the default values set for the parameters if
the tool has `ToolIOCase`s and takes them into account when figuring out
the output type. I've built it with horrible Java reflection magic to
avoid having to specify the default for params twice, which will make it
impossible for the defaults to disagree with each other. This just runs
once at startup so there's negligible performance impact.
The change is easily tested with Change Parameters, which is just
`add-password` behind the scenes but with the password params omitted
(so Change Password is always PDF->PDF, never encrypted like Add
Password).
Also (somewhat hackily) fixes a bug I noticed where saving a Change
Permissions step then leaving and returning to the pipeline will cause
the step to be reloaded as Add Password. I've added a system to
disambiguate tools which share the same endpoint (which is only these
two currently).
## Currently
<img width="455" height="135" alt="image"
src="https://github.com/user-attachments/assets/c8867d6a-599b-4f21-a2db-4a1b6ac22d73"
/>
## Now
<img width="415" height="122" alt="image"
src="https://github.com/user-attachments/assets/bd8d1bab-f00a-421b-8c91-5af0d2ad5335"
/>
# Description of Changes
Nightlies keep failing because the Playwright tests only run on Chrome
in PRs. This PR changes it so that we run all 3 browsers in all
(frontend) PRs so we catch these things before they merge in. They run
in parallel so it won't take any more time for the CI to finish.
# Description of Changes
Continued effort towards removing all uses of the any type in our
frontend code (last PR was #7326). This PR fixes 7 more folders and
removes them from the exclude list. All of them were localised within
the folder in the exclude list so again were pretty easy to fix.
# Description of Changes
Smaller scope than #6689 to try and get this finished.
Replace ESLint and dpdm with Oxlint, a TS linter written in Rust so its
performance is dramatically better than the existing tools we use.
## Speed improvement
- Current ESLint run: 13.76s
- Current dpdm run: 3.59s
- Total time: 17.35s
- New Oxlint run: 0.90s
So Oxlint is about a 20x speed improvement.
## Differences
When I last tried to do this, we could recreate our rules identically
with Oxlint, but that's not true any more. Oxlint has no current
equivalent for ESLint's `no-restricted-syntax` rule, which we were using
to ban usages of `<button>` and stuff in specific components to try and
encourage them to use our shared UI. This is a very recent addition to
our linting config, and personally I'm willing to drop it for now at
least. We can still ban specific imports in files, so the files which we
were trying to enforce shared UI will still ban directly importing
Mantine, so that'll probably be most of the cases still caught, but I
think there are other ways we can encourage using the shared UI beyond
just using the linter for it.
I did try building a custom TS rule for it and it only slowed it down a
tiny bit (it took 1.1s) but it had to be built on an unreleased alpha
API which just sounds like a maintenance headache we don't need to deal
with for a rule that we don't really need.
# Description of Changes
Continued effort towards removing all uses of the `any` type in our
frontend code. This PR fixes 10 more folders and removes them from the
exclude list. All of them were really simple fixes.
# Description of Changes
The restructuring of the frontend PR (#7062) has highlighted a couple of
issues with the source which are currently hidden due to the structure
of the code. This PR fixes the issues in Watched Folders:
- There's a circular dependency in TS code
- The `.gitignore` file excludes the watched folders dir by accident
# Description of Changes
Mac builds in PRs are not currently signed, which means you can't run
them when downloaded. This restores the functionality so that Mac builds
are always signed.
# Description of Changes
Unused translations test takes ~8 seconds to run on my computer with no
contention, but when my CPU is under heavy contention, it often takes
>20 seconds and occasionally goes over the 30 second timeout. This PR
changes the test to use [a trie](https://en.wikipedia.org/wiki/Trie) to
more efficiently search for the strings, taking the test down to ~1.8
seconds. I've also increased the timeout for the missing & unused
translations tests for belt-and-braces.
# Description of Changes
The `task backend:test` command automatically spawns new Java processes
in the dock on Mac as it runs, which takes the focus away from whatever
the developer is doing at the time. This is because there's missing a
missing `headless` tag in the `build.gradle` file (the tests don't spawn
or require any windows, so they run fine headless).
Also adds a `task backend:test:force` rule to run the tests without
cache because the cache was getting in the way of testing this.
<img width="175" height="98" alt="image"
src="https://github.com/user-attachments/assets/d4524959-9c25-4513-bf34-9fd48310c4d3"
/>
# Description of Changes
Move the logic for validating params into global exported functions and
attach them to the operation config so that we can access them elsewhere
so things like Pipelines know whether the tool has been configured and
provide a warning if not.
<img width="1264" height="503" alt="image"
src="https://github.com/user-attachments/assets/f320579c-6c08-4c66-a999-015d46df1b33"
/>
# Description of Changes
Storybook files aren't currently being type-checked, but they should be.
A bunch of them had a dodgy import in them (which didn't affect anything
because it was just an `import type` but still worth fixing).
# Description of Changes
Change pipelines so that sources and triggers are grouped into a list of
inputs, so you can have a different trigger for each source in the list.
This is necessary because triggers are not universally supported by all
source types. If you wanted to have a pipeline pull from both a folder
and an S3 bucket, the current system allows you to choose "Folder Watch"
as the trigger, which will either do nothing or crash when it's paired
with the S3 bucket.
I've got reservations about actually allowing different triggers for
every source because it allows for user workflows that I don't believe
exist, like "I want this folder to be polled every minute and this other
one to be polled every hour, but they should run the same tools and
should output to the same place". Because of this (with agreement from
Connor, Anthony and Matt) I've changed this PR to artificially limit
pipelines to having 1 input & output at this stage. The backend is still
shaped to support multiple inputs & outputs so it should be trivial to
re-add support for them in the future if we decide we want to, but the
UI can be much simpler and easier to understand with just 1 input and
output.
<img width="1262" height="521" alt="image"
src="https://github.com/user-attachments/assets/809e6803-9f99-436d-9aeb-52dddf0906ff"
/>
# Description of Changes
Currently, desktop PRs only build on Linux, which none of the core
maintainers currently use. Change it so that desktop PRs build Mac and
Windows, so core maintainers can test the built version.
# Description of Changes
The `pre-commit` tool to sort the translations is really slow. It took
~40 seconds to run because it's using a parser which attempts to save
all of the formatting data from the Toml. Our translations toml is
pretty much entirely formatted anyway, so there's no point in trying to
preserve any of that data. The only thing we lose is 5 comments, none of
which are needed anyway and only appear in the US translation file. By
switching to Python stdlib `tomllib` reading and `tomli-w` for writing,
we can make the Toml formatting job take 2.11 seconds, where it used to
take 39.78s. The whole pre-commit job now takes 4.58 seconds.
# Description of Changes
<img width="1270" height="487" alt="image"
src="https://github.com/user-attachments/assets/64894e2b-aab9-42ab-96c2-2c11ba427b52"
/>
Change policies to point towards a source for its output instead of a
dynamically defined output location for the pipeline. This allows for
easy reuse of outputs in different pipelines and makes it impossible to
break complex pipelines by accidentally updating the source but not the
output and vice versa. Also makes outputs a list to match the inputs, so
it's possible for a pipeline to output to multiple locations.
We should consider whether we want to continue calling these Sources
since they're now being used as both inputs and outputs, but that
decision is beyond the scope of this PR.
Also updates the existing S3 DB migration script and adds a new one to
migrate to the new schema. Neither of these scripts are possible with
SQL since it involves parsing and restructuring JSON. I've updated them
so that they only ever run once on startup and mark themselves as
completed.
# Description of Changes
Adds implicitly defined folders to the list of locations that folder
sources can look in, including the legacy watchedFolder folders, and the
server storage location (if enabled). Also adds a settings UI for
defining the list of allowed folders instead of having to manually edit
`settings.yml` (please excuse the styling, that's the standard styling
of the Processor, hoping it gets fixed by one of the styling PRs).
<img width="888" height="786" alt="image"
src="https://github.com/user-attachments/assets/cf6d0705-adcf-463c-8e80-6901a652068b"
/>
<img width="1103" height="713" alt="image"
src="https://github.com/user-attachments/assets/b7244592-249a-4149-994f-3a2b750f25f9"
/>
# Description of Changes
Paired with https://github.com/Stirling-Tools/Stirling-PDF-SaaS/pull/320
Fixes the following bugs we found when testing the SaaS release:
- Existing users couldn't join teams - this was because they were the
last leader of their team, so it'd be left orphaned). Users now have a
'home team', which can have no members if they join another team, but
they can then go back to it later.
- Existing leaders didn't have unlimited seats - `saas_teams_extensions`
had no row for them, so the app fell back to `max_seats=1`. The
migration script fixes it.
- Members without Processor access could still access the Processor - It
was just checking "Are you the leader of **any** team", instead of the
user's active team.
# Description of Changes
There's currently a column size inconsistency between the SaaS v3 DB and
the main Java code which causes the backend to fail to start up when
connected to a fresh DB. This is because the column previously was width
255, but now it's officially width 50, but the Java type is still
implicitly `varchar(255)` because there's no length attribute. If it's a
fresh DB, Postgres throws an error that it can't expand the column (this
doesn't error on an existing DB because the column is already wide
enough behind the scenes).
Co-authored-by: EthanHealy01 <80844253+EthanHealy01@users.noreply.github.com>
# Description of Changes
Removes the feature flags for enabling policies on both the backend and
frontend. We shouldn't be releasing another self-hosted release that
doesn't include policies, so it makes sense to do this now. Builds that
don't have the Processor will just not run policies because they won't
have any. Beyond that, the API should always be available, but checks
whether the user actually has the entitlements to run policies (whether
they have credits/a payment method available)
# Description of Changes
The policies stored in the DB are currently encrypted at rest, because
one version in the past included S3 keys. These are now stored properly
in the credentials system and I've manually removed the only policy that
used S3 (it was very recently released). Since there's no S3 (or other)
credentials in the policies stored JSON now, we might as well just
decrypt them. This PR pairs with #7034 to fix the issues - #7034 makes
it resilient to crashing when attempting to load encrypted JSON that's
been encrypted with the wrong key, and this makes it so if it does load
any encrypted policies, they'll be re-saved decrypted, so we should have
a vanishingly small number of encrypted policies over time.
# Description of Changes
The Policies page and all the frontend logic for running Policies is not
making use of the bidirectional type mappings that we now have to safely
convert from frontend to backend param models and vice versa. This
changes the way we track the types throughout so we use the mappings
properly.
Because of this, the Add Watermark settings in Policies now actually
pre-populate with the defaults instead of with nothing like they
previously did.
<img width="791" height="725" alt="image"
src="https://github.com/user-attachments/assets/cbdf4ae0-35af-4792-bf64-89216e48d304"
/>
# Description of Changes
Some of the tool settings make use of editor preferences indirectly, but
the Processor never gets that provider, so it crashes when trying to
load them.
# Description of Changes
Redesign S3 connections based on feedback from #6948. Also redesigns the
UI for Sources to make them more like the Pipelines page which improves
UX quite a bit. There's still plenty more UI/UX work for Sources and S3
but moving in the right direction.
# Description of Changes
Policies can currently throw when calling redact:
<img width="1186" height="824" alt="image"
src="https://github.com/user-attachments/assets/bdcc09fe-5bf4-4b0a-b119-bcc33c98c7f2"
/>
Policies really need to be updated to properly make use of the new
bidirectional mappings for this, but this will hopefully fix it for now.
# Description of Changes
Adds Editor source permanently available in the Sources list. Excludes
it from the Pipelines list of available sources currently because it's
not a real source on the backend, so attempting to connect to it causes
an error. It'd be nice to extend in the future to be able to set up
policies in the editor from the pipelines page, but this'll do for now.
# Description of Changes
Replaces the `.stirling/done` folder and its friends with a ledger in
the DB which tracks which documents have been processed. This should
scale dramatically better since it's just a few bytes being written for
each PDF processed, rather than each PDF being duplicated and held in
the folder forever. It's designed to work with the current folder
source, but also with S3 buckets and other sources in mind - each source
will define its own strategy for ensuring it knows whether the documents
have had policies run on them or not, and they all get written to the
same ledger.
# Description of Changes
Disallow SaaS guests from accessing the portal. One day we might want to
make this better so they can go there but then have to sign up before
doing anything useful, but this is the easiest way to disallow it for
now.
# Description of Changes
The Change Metadata tool was missed from the bidirectional mappings
added in #6867. This PR adds it to the list of supported tools.
# Description of Changes
Followup work requested in review of #6867. Currently, there is nothing
enforcing that the endpoint chosen in the tool config is the correct
mapping for `toApiParams`, so theoretically it's possible for a tool to
be set up to call an endpoint with the wrong API params for it. There's
also nothing currently enforcing that `toApiParams` and `fromApiParams`
are compatible with each other (using the same types). This PR changes
it so that instead of creating the config object directly, tools create
it via a generic function, which enforces that all of the relevant
mappings are using compatible types.
# Description of Changes
Modernises the codebase and gets rid of warnings where Node complains
that it doesn't know what type of JS it's supposed to be reading on
`.js` files. We might as well update everything to just use correct JS
syntax instead of keeping with some files having Node-specific imports.
# Description of Changes
A few changes to improve things in the engine:
- Changed the PDF to Markdown code to be a real tool in Java, to remove
the need for the `pdf_ingest` code, which looked a bit like an agent but
wasn't behaving as an agent. It's now just covered automatically by the
edit agent.
- Noticed that 0-parameter-endpoints were previously being ignored by
the `tool_models` generator, so some tools which require no params were
being mistakenly excluded.
- Removed tools which currently never succeed like Add Stamp, Cert Sign,
and Overlay, because they require the supporting files to be sent in a
different location in the API call, which we don't currently do.
Ideally, we'd add proper support for this, but we're better off now
removing support for these tools rather than just have them crash. We
can re-add these tools in a future PR properly.
# Description of Changes
Fix https://github.com/Stirling-Tools/Stirling-PDF-SaaS/issues/281. Add
generated backend API mappings to the frontend code, and the logic to
convert from a backend API to frontend parameters objects.
Previously, it was impossible to tell if changing the backend API would
require a change to the frontend to support it because the frontend had
no static type information about the backend API. This PR adds
autogenerated tool API types to the frontend (in `toolApiTypes.ts`) and
adds explicit typed mappings between the frontend parameter types and
the backend API types, so theoretically the type checker should be able
to catch issues when changing one puts us in an invalid state with the
other. During development, it pointed out several inconsistencies that
we have between the frontend and backend types, some of which were
genuine bugs, and others were only happening to work because the backend
is more permissive than its API claims to be.
This also unlocks the ability for us to render the frontend settings on
saved backend API structures, which we've previously had to avoid doing
because we had no reverse mapping.
# Description of Changes
Fixes intermittently failing tests (and replaces one that wasn't useful
in its previous state) and also adds a CI check to warn if there are any
Playwright tests which failed on their first go and succeeded on
retries, to hopefully help find intermittently failing tests more
quickly and avoid them being merged in the first place.
# Description of Changes
Playwright tests currently fail in Firefox and Safari because of
inconsistent behaviour across the browsers. This is causing the
nightlies to fail every night. This PR fixes the test behaviour to work
consistently across browsers (most of the issues were to do with the
tests opening the file picker, which was being automatically suppressed
in Chromium, but not the other browsers).
# Description of Changes
We don't have any strong reasons to keep the Portal as a separate Vite
app, and it needs access to so many things from the Editor that it no
longer makes sense to keep them separate. This PR moves the Portal code
to have direct access to the Editor code and gets rid of the shared
folder.
# Description of Changes
Fix#6801, along with fixing policies on desktop, which would attempt to
download policy outputs from the local backend instead of the server,
where they actually live. I've changed the policies logic to maintain
the same backend for the file retrieval as it used for the policy
running, so when we support running policies locally, it should still
work correctly.
# Description of Changes
Continued effort to expand linting scope to ban the `any` type in our
codebase. This PR pulls in a lot of subfolders into the linting scope,
because the excluded list was getting short enough that it was feasible
to move a layer down. I then fixed all the trivially fixable `any` type
violations in the subfolders, which just required local changes to the
one file. The aim of this PR is more to expand the scope to all the
folders we can that already avoid `any` types, rather than actually fix
violations.
# Description of Changes
We can't convert to TS7 completely yet because it lacks the TS API, so
ESLint and some of our scripts don't work, but we can do [what the TS
team suggest and run TS6 and TS7
side-by-side](https://devblogs.microsoft.com/typescript/progress-on-typescript-7-december-2025/#compiler).
When we do that, we take the `task frontend:typecheck:all` job from ~76s
to ~13s, and everything else continues to work as it did before.
I've set it so that CI will still use TS6 for the time being and locally
we use TS7 out of an abundance of caution because CI time doesn't really
matter but local time does. I do think it was a bit pointless doing that
since the TS team claim the type checking performs identically, but we
might as well have it like that for now. If it happens to go badly
locally for any devs, they can use `CI=true task frontend:typecheck` to
revert to use TS6 trivially.
# Description of Changes
[Our nightlies have literally never passed
before](https://github.com/Stirling-Tools/Stirling-PDF/actions/workflows/nightly.yml).
As far as I can tell, that's because the frontend was never being built,
so the Playwright tests would just never start up.
I've forced a nightly run from this branch, and the Playwright tests
still fail, but for legitimate failures now. It's a separate job to
track down why they're actually failing, so I'm leaving that for
followup work.
# Description of Changes
Redesign policies backend to treat sources a lot closer to how the
frontend imagined them working (they're persistent now and have an API).
Then connect the portal to the sources when mocks are off to allow for
source creation in the UI. It's not particularly useful to do that right
now because there's no policies UI, but I've tested manually that
sources set up in the UI are usable by policies created via the API.
I had to change the portal so that when mocks are off, it doesn't just
hard crash when attempting to connect to all the backend APIs that don't
exist yet. It'll still log the errors, but just continues on rendering
the UI now.
I also changed all the policies backend APIs to be gated behind a flag
instead of behind the SaaS profile. This is because we haven't yet got
the payment model sorted, but we're going to need this stuff running
self-hosted to be able to test it locally.
# Description of Changes
- Change the nightly build to not sign any of the desktop builds, since
we just care about the compiled code. The restored code will still be
signed dependent on the OS in the PR builds.
- Change RPM Linux to use zstd for compression because the one it was
using runs really slowly, and the Jar is already compressed so it makes
basically no difference (arguably we shouldn't compress at all)
- ~Switch to consistently use Depot for Docker caching to stop filling
up the GHA cache and evicting the Rust cache~ Decided against switching
to Depot because we're probably doing another PR to remove Depot
altogether in the near future
# Description of Changes
Fixes one of the main causes of `any` typing left in tools, the way that
we register tool parameters in the registry. Currently, it just accepts
tool params via `any`, but instead we can explicitly change them to
`Record<string, unknown)`, so on the way back out they can more safely
be cast back to their correct type when known.
One consequence of this is that I had to redesign the way we
special-case the Convert tool, which previously was a different shape
than all the other param types. Now it's just got optional parameters on
it, which isn't quite as type-safe as before, but it does mean all tools
are a consistent shape now, which I think is worth the tradeoff.
# Description of Changes
TS6 introduced backwards-incompatible changes which affected us a little
bit. Other than that, I don't think it significantly changes things for
us, but we will need to deal with these breaking changes to be able to
upgrade to TS7 (the version written in Go, so dramatically faster), so
I'd rather do the work now before TS7 actually releases.
Main things I've done:
- Removed the use of `baseUrl` in the `tsconfig.json` files
- Explicitly provide the `node` types where needed
- Explicitly reference the un-referenced but required Google API types
- Dropped the installation of `madge` which we weren't using and wasn't
directly compatible with TS6
- Updated the `i18next` packages for explicit TS6 compatibility
- Explicitly override `tsconfck` to force TS6 compatibility since we
can't upgrade it. We're only using that for `vite-tsconfig-paths` and it
all still seems to work fine, so I don't think this is an issue. I think
we can theoretically drop `vite-tsconfig-paths` when we upgrade to Vite
8 ([because it supports
`paths`](https://v8.vite.dev/guide/features#paths)), but that's a bigger
job than I want to do in this PR
# Description of Changes
Refactor frontend auth to the shared folder and hook it up to both the
portal and editor so they share the same system. Also adds various tasks
to help run the portal, including `task dev:portal` to spawn the portal
with the backend, and `task dev:portal:proxy` to spawn the editor,
portal and backend, and a reverse proxy (at localhost:3000) to allow you
to use both at once to simulate how this will actually be deployed,
allowing you to check whether the seamless transition between the two
actually works.
# Description of Changes
> [!note]
> GitHub absolutely mangles the diff unless you change to ignore
whitespace changes
This page has never had the Enter key bound to the Change Password
button:
<img width="673" height="745" alt="image"
src="https://github.com/user-attachments/assets/7a1b06f0-2945-4270-a795-f799ec556c12"
/>
This PR changes the modal to be properly wrapped in a form so key
commands work correctly on it.
# Description of Changes
Rust cache added in #6732 never fired because `main` builds don't
include building the desktop apps. We could build them on `main` builds,
but that's fairly expensive, so just build them on nightlies instead to
warm the cache for any desktop PRs the next day
# Description of Changes
The Tauri jobs are very slow, especially the Linux ones, which can take
>1hr to build all the necessary code. A lot of that is because of the
actual Rust compilation, which isn't cached at all as far as I can tell.
This introduces a cache step for the Rust dependencies, so PRs will just
reuse the compiled Rust from the last build of main (if it's safe to do
so).
# Description of Changes
Continued effort to remove the remaining uses of the `any` type from our
TS code. The vast majority of these uses that it cleans up was just
catching errors as `any`, which are pretty simple to fix. I couldn't
completely remove the `any` type usage from `core/tools` because there
were cascading issues from a couple of the files in there (most notably
Automate) but still, moving in the right direction.
# Description of Changes
#6727 introduced frontend code which goes against the architecture, so
this PR re-implements it in the architecture properly, along with
another bad Tauri check that I found in the source. I also updated the
`AGENTS.md` file to use Claude's "read this file" syntax to try and
force AI to actually read the file instead of just suggesting that it
does it.
# Description of Changes
Add message describing the most common tasks when running `task` with no
arguments. I think this should help newcomers because `task --list` is
massive at this point and nobody's going to read through it all. Let me
know if you think any other commands should be in the default message.
# Description of Changes
The `pre-commit` commands in this repo are inconsistent with the rest of
the dev workflow, as they are impossible to run through Task and they
can cause CI to fail with no way for a developer to run the `pre-commit`
scripts after they've failed. This PR adds `task pre-commit` (and `task
pre-commit:fix`) and then hooks up the existing `pre-commit` hooks and
CI to call the Task rule, so if developers are using pre-commit hooks
then they should still work, but they're also runnable without using
pre-commit at all.
I think it'd be worth reviewing what we're actually running at
pre-commit in the future because I'm not entirely convinced by all of
the scripts that we are running, but this should at least make what we
have properly enforced and usable by all devs.
# Description of Changes
Currently, `task dev` explicitly calls the backend with
`AIENGINE_ENABLED=true` even though it isn't being spawned, so you just
get a dead FAB in the UI. This PR fixes it so that the engine will only
be enabled for tasks that will actually spawn the engine.
It also fixes a bug with the chat which makes it unusable locally. The
API path was not going through `apiClient` so for local dev you end up
with `//api/v1/...` which is not a valid path, so you get CORS errors
when trying to connect to the AI engine.
# Description of Changes
Redesign policy running so the server is in charge of policy IDs and
running, to make it impossible to have the frontend miss the results.
This solves a minor bug that we currently have in policies, where if you
load a file and then refresh while the policy is running, you'll never
receive the outputted file.
# Description of Changes
One of the Playwright tests is flaky, despite several attempts to fix it
before it made it into main. This disables the test for now so a
followup PR can try to fix it again.
# Description of Changes
Fix Playwright failing test in SaaS (I think this is my third attempt
now so who knows if this will actually fix it for real this time, but
hopefully it does)
# Description of Changes
> [!warning]
> **Do not** squash this on merge. It should be merged via a merge
commit
Fixes conflicts in `pgvector_store.py`.
Also since codespell is failing, add comments to ignore the errors in
`sync_en_us_spelling.py`
---------
Co-authored-by: Ludy <Ludy87@users.noreply.github.com>
Co-authored-by: Anthony Stirling <77850077+Frooodle@users.noreply.github.com>
# Description of Changes
* Remove complex port selection logic from `engine.yml`. It's
inconsistent with the frontend & backend task files, and caused issues
with Docker, which have been worked around but would be simpler to just
get rid of the problem altogether
* Fix Ruff formatting of Python script
* Remove payg tests which are failing and have drifted too far from the
implementation to save directly
