I’m curious how this increased throughput happens.

You’ve accurately stated that AI isn’t as rigorous as a trained attorney. Doesn’t that mean that every single datapoint must be confirmed by a human?

How is that quicker than just using a human to read the content and make the call? Data entry savings?

Yes, remember that these are effectively random PDFs in various different designs and formats, some of them not editable or even OCR'd.

It took a human attorney 20-30 minutes on average to manually copy-paste data from these PDFs into a spreadsheet (while also fixing any errors they found in the document and re-checking for quality).

Now, the AI copies everything into the spreadsheet in a small amount of time, and then the human reviews it. It takes maybe ~5-7 minutes to scroll to the appropriate pages in the document, read the lines vs the spreadsheet, and make corrections. So you've gone from 2-3 items an hour to ~8-10 items an hour.

Maybe you could pay someone to develop an OCR/ML application that could do this. But that project would never be profitable, even with the time savings. At the cost of a couple Claude subscriptions, it makes sense.

Does the human find enough bugs that they stay on guard, or just rubber stamp everything without really looking at it? It’s hard to stay vigilant when stuff looks plausible.

Complete speculation: you could instruct Claude to hide one random mistake in every document.

And Claude should write down the mistake in a sealed envelope, so it doesn't make into the database.

A review that doesn't find the mistake counts as invalid.

This is what bag scanners at airports do - the hit rate is so low and the job so boring the software projects fake contraband onto the imagery. Fail to spot the knuckledusters and expect a chat with the manager.

Don't forget that humans have a not insignificant error rate when copy/pasting or copy/typing data.

And it's possible to run each document through the LLM pipeline multiple times, using different models and/or prompts each time, to check for errors and inconsistencies. That will take more time and cost more, but it can reduce the error and hallucination rate significantly.

I'm doing some public court records processing for bankruptcy cases (interested mostly to seek out corruption in big national cases), and yes, the "variousness" of random PDFs is exactly the issue. Trying to get the cost for a whole case down to a minimum.

Sample is around 300 court dates, shy under 1k files.

At best I'm building a claude skills file.

Nobody working there has the requisite experience to do this in a reasonable amount of time. These are not particularly tech-savvy folks, Claude use aside.

> These are not particularly tech-savvy folks, Claude use aside.

The difference between a tech-savyy person, and a non-tech-savyy person has always been mostly in the later's head, but this is even more true now that we have pocket assistants who can answer pretty much all of our questions in a language tuned to our level of understanding.

Yes. And it might not even be worth it, as the AI agents gets cheaper and cheaper.

Keep in mind that the task is fixed, so as the frontier of AI advances, you can switch to a cheaper trailing edge system and still get the same or even better performance for this task.

Thanks for sharing the details. Does the attorney check that the AI copied the data accurately? Or is it just assumed to be correct?

Your experience mirrors my own. AI is great for parsing data that can take up a huge amount of time. My only concern is whether or not it’s done accurately. I wouldn’t use it for anything where mistakes cause serious consequences.

You don’t need a trained attorney to schematize data. The LLMs are used to make the data easier to understand and manipulate.

They'll also hallucinate and change meaning in the process of extraction and "schematization"

Not necessarily. Depends how you use it.

"Write a python script that breaks down this PDF by X feature" would not hallucinate anything in the PDF. Certainly you could trivially double check that all text in the extracted JSON was in the text layer of the PDF.

How much experience do you have with LLMs exactly? It would be consistent with my experience if Claude stuck in a line of python that just emits a JSON literal with no justification, potentially buried in a large program where an untrained person might not notice it. I don't even trust them if the output consists of structured data paired with source images from the PDF, because I've experienced LLMs fabricating the source rectangles to match the output. I only use tools like this by asking for programs, because as you note LLMs are good at that, and the verification process consists of tool calls to legitimate PDF manipulation tools so I have some confidence everything is above board. Even then I only do this for hobbies, not anything that matters.

Lawyer here. I used to trust Claude as hallucinations are near non-existent now. However for large volume tasks such as due diligence exercises, they still happen. We also tried Legora's tabular review, there were also numerous halucinated provisions in our due diligence exercise.

Is it possible to catch those hallucinations using another LLM with a strong fact checker prompt with sources provided in output for human validation?

Junior associates hallucinate too...

And when they do, you can train them or fire them, and they learn not to do it.

LLMs change not a whit, and there's no one to take responsibility for the failure (and thus no way to fix it).

As the new variation on the old theme has it, "A computer can never be held accountable, and so very many people are trying to get them make management decisions."

> LLMs change not a whit, and there's no one to take responsibility for the failure (and thus no way to fix it).

LLM output is nondeterministic and humans take responsibility for the failure the same way they take responsibility of a photocopy is too dark.

> humans take responsibility for the failure the same way they take responsibility of a photocopy is too dark.

You mean, they notice it's too dark right after making it, change the settings, do it again, and give you the good copy?

Because yes, that's my experience of humans.

You can’t train people to never make a mistake, particularly when doing highly repetitive work like this. You must build your systems to account for that regardless.

For sure.

But they do learn and improve.

The models don't (yet).

The models improve in the sense that GPT 5.6 succeeds at things GPT 5.5 fails at.

It might be that the models have been improving in this sense faster than a human child improves.

Yes, exactly. Humans are non-deterministic as well, just in different ways. A tired human can make all sorts of errors for example, regardless of how much training they've had.

You can scold juniors and they will learn. You can't scold Claude.

Surely the rate of improvement in new LLM models is the equivalent mechanism?

You can scold Claude. Just doesn't make a difference.

... until you hit Claude's risable "model welfare" protection.

That's a recipe for disaster in my experience. I tried it (with Claude) on a simple tabular bank statement PDF, and it transposed two amounts, placinh each against the other's description. And the bot assured me the result was cotrect. The chance of a human checker catching such corruption is low.

Interesting. Did that PDF have a text layer or did you ask Claude to OCR it? If the latter I'm not surprised at all.

Text layer.

But it gets it right like 99% of the time so human attention can be put towards catching the 1%, not entering data from one table to another and then catching that human’s mistakes.

Doing similar-ish things with Claude, it's helpful to have something to ground it.

For instance, if you can say:

"Refer to the database schema in x.sql as your source of truth for the database structure we want to import into*. Do not invent data, tables or columns that do not exist. Carefully match all output against this database schema and do not create output that doesn't exist if it does not match the schema, simply skip it."

You will end up with a far better result in my experience.

Gotta treat it like a child.

> Gotta treat it like a child.

"Sorry for that, Your Honour, but we gave that case prep work to a child."

They do, sometimes. That's why the review still has to happen.

But now it's comparing already filled columns on a spreadsheet, not copy-pasting every single thing from an (often uncopyable) PDF.

> But now it's comparing already filled columns on a spreadsheet

... with a PDF, right?

> not copy-pasting every single thing from an (often uncopyable) PDF

Obviously the PDF is copyable, else your bot would not be copying it.

Reviewing something takes less time than producing it.

I...don't think that is universally true.

Few things are. It also doesn’t require standards. “Stamp this diff” culture is everywhere even before AI. A stamp is literally easier than anything else.

Whether that is useful measurement I suppose depends on the circumstances.

Yes, but empirically in this case, it seems to be true.

Little known fact - Vietnam modernized their Public Security apparatus in the late 1950s with help from GDR’s Stasi.

Vietnams surveillance system, population database and use of biometrics were all adopted from the GDR.

As a Vietnamese, that sounds meaningless because Vietnam never had much of a "surveillance system" nor an effective population database well into the 2000s. I have no doubt the effort was there for them to control people, but a complete lack of competency invalidated it.

Vietnam has had a security apparatus all the way down to the neighborhood block level since the 1950’s. It was low tech and relied mostly on human surveillance but it did exist.

It has collected biometric information for a while and now has facial recognition in place for every single Vietnamese at this point. Combined with the camera coverage in Vietnam now I wouldn’t be surprised they can track anyone they wanted.

Public security cameras are not widely deployed, like, at all. However the police state in Vietnam (the current president came from the same position Putin held before presidency) has adopt this amusing practice of showing up & demanding security footage from private houses' cameras facing the street (which, turns out to cover much more than surveillance cams could ever).

More insane is them also passing a law that forbid usages of the same footages for citizen journalism purposes. This came out after someone posted a footage from their home cam recording that controversial accident when a congressman hit and kill a high-schooler unprosecuted. You can look it up.

I’ve been paying attention to the incident you speak of since it’s so interesting to see how an authoritarian system deals with it.

I was quite amazed when the father of the teenager made a statement about “maintaining harmony”. It’s right out of the stories you’d hear about life in the USSR.

The error that people make when criticizing card fees on merchants is that customers pay the cost. That’s not necessarily true.

There are benefits for merchants to accepting credit cards. More impulse buys, buying more per transaction. An owner may choose to absorb the merchant fee because the increase in sales (even at the lower profit per unit) means higher profits overall.

This is a solved problem.

Let people defer increases until their house is sold. This what many states do.

It’s seems insane that we “protect” people whose property taxes are going up because their home is getting so valuable

It's also a solved problem by not deferring anything unless that person is past working age and low income. Letting people sit on wealth that they cannot pay a wealth tax on is generally bad for everyone with less wealth.

It’s hard to take comments like this serious with all the hyperbole.

But regardless. Polls of your average citizen aren’t worth a whole lot considering most people get their information from social media.

What's hyperbolic? The trust thing is borne out by repeated polling of which countries people trust. The current US administration is openly profiteering from governing.

I do think it'll take a few years before Americans wake up to how badly their reputation has tanked around the world across successive governments of either party in the recent decade.

Greenland, Iran, Falklands, Ukraine, Gaza, Taiwan, all these things probably feel like minor missteps in the US news cycle but they have had a major impact.

It's not quite a mafia state yet. We've yet to see Trump opponents fall out of windows.

I think Americans are unwilling or unable to see just how much foreign attitudes towards the US have shifted.

It can be something as simple as an analytical reference.

If you’re testing for dimethyl mercury in say fish, you need a standardized sample of dimethyl mercury to run.

These references are highly dilute and thus less hazardous. But some labs prepare their own references rather than buying them.

That requires working with pure dimethyl mercury.

The hazard is manageable as long as you’re prepared. Chemists work with lots of hazardous chemicals and it can all be managed with the right preparation.

I am slightly over three thousand years old.

Who is talking about illegal immigration? This thread is about legal visa holders following the law.

They aren’t legal any more if their visa expires?

I’m an immigrant to the US. I knew the risk that at any time the rules could change and I’ll be headed to my home country.

That was an isolated incident triggered by local police. Not an active nationwide policy to deploy immigration officers to all migrant protests

Nope. There was another protest where the police asked the protestors status. They then called Immigration Canada to verify.

You literally have the cops in Canada working hand in hand with federal immigration enforcement. No politicians crying about it. No protests by Canadians.

Just the enforcement of common sense law

Did you read the article? They postponed all appointments.

The pro maga crowd has learned to take a less obvious, but just as callous approach to commenting over the last 6 months. They are trying to subtly slide the overton window in their direction.

“Nuh uh you.”

Quite the rebuttal.

Don’t blame me when someone inserts themselves into a discussion they don’t understand and start throwing insults

You sure? Canada's economy is struggling as is. The healthcare system is holding on by a thread. You ready for unemployment to double? If there is a significant recession, tax reciepts go down and healthcare spending gets further cut.

Such exaggeration.