Interesting idea...but could a crawler not just incorporate some AI like LLava2 or convert the SVG to a JPG and use OCR to get the email addresses out?
It just seems like this adds a couple of steps to existing crawler scripts.
My (peripheral) experience is that it is much cheaper to get data in than to get data out. When you have the amount of data being discussed — "Enterprise data. Going back years." — that can get very costly.
It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).
AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/
> Q: Can I export data from AWS with Snowmobile?
>
> Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more.
"Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction."
There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck!
I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere.
I personally dont care if the docs are crap because usually the first thing I do is update / actually write the docs to make them usable.
For a lot of techs though crap documentation is a deal breaker.
Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written. Quite frequently though its outsourced development agencies that refuse to write it because its "out of scope" and a "billable extra". Which I think is an egregious stance...doxs Should be part and parcel of the project. Mandatory.
I agree that bad documentation is a serious problem in many cases. So much so that your suggestion to write the documentation after the fact can become quite impossible.
If there is only one thing that juniors should learn about writing documentation (be it comments or design documents), it is this: document why something is there. If resources are limited, you can safely skip comments that describe how something works, because that information is also available in code.
(It might help to describe what is available, especially if code is spread out over multiple repositories, libraries, teams, etc.)
(Also, I suppose the comment I'm responding to could've been slightly more forgiving to GP, but that's another story.)
> Quite frequently though its outsourced development agencies that refuse to write it
It's also completely against their interest to write docs as it makes their replacement easier.
That's why you need someone competent on the buying side to insist on the docs.
A lot of companies outsource because they don't have this competency themselves. So it's inevitable that this sort of thing happens and companies get locked in and can't replace their contractors, because they don't have any docs.
Unfortunately it’s also possible that e.g the company switched from share point to confluence and lost half the entire knowledge base because it wasn’t labeled the way they thought it was. Or that the docs were all purged because they were part of an abandoned project.
> the first thing I do is update / actually write the docs to make them usable.
OK so the docs are in sync for a single point of time when you finish. Plus you get to have the context in your head (bus factor of 1, job security for you, bad for the org.)
How about if we just write clean infra configs/code, stick to well known systems like docker, ansible, k8s, etc.
Then we can make this infra code available to an on prem LLM and ask it questions as needed without it drifting out of sync overtime as your docs surely will.
Wrong documentation is worse than no documentation.
Just to be clear, after I (and a few others left), they moved everything entirely to the cloud.
Even with documentation on the hybrid setup, they'd need to get a new on-prem environment up and running (find a colo, buy machines, set up the network, blah blah).
"Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written."
I can always guarantee a stream of consciousness one note that should have most of the important data, and a few docs about the most important parts. It's up to management if they want me to spend time turning that one note into actual robust documentation that is easily read.
Man, I started building sites in the mid to late 90s as a kid, for shits and giggles, then it became trendy to hire a "whizz kid" and people paid me to build garbage...web design business was so easy back then..."Your Dad says you know how to build websites? If I give you £500 will you build me one?"...hell yeah I will, we can have a big fucking banner, some scrolling marquees..."sure kid, whatever just make it look good"...no meetings, no wireframes, no bug trackers just here is £500 build me a site kid...by the way I'm a plumber. Here is my brochure...make it look like that.
Then your work began. First stop, looking for gifs of spinning spanners, cartoon dudes with hard hats on. Then you'd get the domain and hosting, put an "under construction" page up...man good times. I wish I could just "build a website" today...but you can't...you have to have 10 people involved bike shedding the fuck out of the fonts instead of having fun...and fucking soul destroying WordPress.
There is. Dropping your phone follows a curve, it falling from 16,000ft it does not. There are more forces at play when you fumble your phone. They aren't necessarily stronger forces though...just more of them. Trajectory and spinning add different forces on top of gravity. There is also the catch attempt that invariably forces the phone down harder and changes the trajectory.
The terminal velocity of an iPhone is around 25-30mph. Past a certain height, it doesn't matter how high up the device is.
Dropping an iPhone out at 16,000ft is probably the same as throwing it off a tall building. I'd imagine a 3 or 4 storey building would be enough to reach terminal velocity.
It just seems like this adds a couple of steps to existing crawler scripts.