(disclaimer: I own a non-profit for submitting requests, education, and advocating the release of digital records)
I seriously recommend that everyone submit a FOIA request for information that they're interested in and make it public. Either locally or federally. You'll learn a lot about the interface between you and your government.
Something to think about - Illinois FOIA allows the submission of code within a FOIA request. For example, as long as you know the schema of a database, you can submit SQL for a government owned/contracted database. The schema is often included in contract specs, but more often requires a request to figure it out. My nfp is working on some projects (through muckrock) to build out a list of schemas across all states' big cities. It's possible, but there's still a lot of work to do!
> you can submit SQL for a government owned/contracted database
I’ve heard of journalists working with government IT folks to craft a SQL query, but not sending code in the records request. Do you by chance have an example letter that uses code?
The idea comes from [1], where like you say, a journalist/requester can work with the government IT folk to craft a SQL query. A big problem comes, though, from the burden and staffing required to write the query. To add to that, FOIA often requires cost estimates prior to starting a request - which, in the case of writing code, can muddle estimations... which have to be paid on the onset. My position is that if the requester understands the schema, then the burden required to write the query is essentially nullified. That way, the only cost would then be the review of the query, which is trivial in comparison to writing one from scratch.
There's a lot of prereq knowledge needed before submitting a request for code. That, and there's generally a potential for penalty against a public body that doesn't run a full search. So in many ways, there isn't really a need for a requester to submit their own SQL from the onset.
It is possible! For example, I recently received [1] a mysqldump of Chicago's "data dictionary" database, which is a collection of database schemas from databases throughout Chicago.
That said - it's not always simple. The big problem is that information being released through FOIA still needs to be reviewed by a person, which can be time consuming and expensive. Especially so when a technical person gets involved. It's often just much easier (for everyone) to request specific information.
The "data dictionary" database dump was only easy to get only because I knew for a fact that the database was already reviewed through a process that matched FOIA's.
Nooo! Think of the untold millions of potential FOIA requests you're eliminating!
At least that's the ideal case scenario. Practically speaking, that would require an in-depth, contextualized review of every single row of every single column in every single schema.
So, the answer is probably yes, pursuant to the "associated costs" field of the request being at least 5 figures, and probably 6.
I submitted a FOIA request five years ago to the State Department. About six months ago I received a partial response which was redacted to uselessness. I’m still waiting for the rest of the responsive documents.
My take away from this experience is that if you aren’t prepared to sue ($450 absolute minimum cost) don’t bother.
Yeah - that's one of the main reasons why I haven't really focused on federal FOIA that much. It just takes too damn long. You aught to try something local - you're much more likely have a more open conversation about the records.
Also - lots of pro bono FOIA lawyers exist to help ease that burden. FOIA covers legal costs if the suit goes in your favor.
When I say "submit", I mean "submit a FOIA request over email with some SQL, then prey that the FOIA officer will google 'SQL' before flatout rejecting your request". :)
Yep [1]! It's pretty damn impressive, but it's by no means a complete list of the data Chicago stores. Which is where FOIA comes into the picture. According to Chicago's (ex) Chief Data Officer, they also have a fairly routine process of scraping through Chicago's FOIA requests for gauging interest for new datasets. In other (somewhat naive) words - the more data-like FOIA requests that Chicago receives, the more open data will be made available.
On a similar note, federal FOIA has some built-in "proactive disclosure" language that makes information publicly available after it's been requested more than three times, or is likely to be requested often or with interest. Not an expert on this piece by any means, but it's something to consider when thinking about open data portals and the like.
Also, check out Data USA's (exceptionally bloated) data portal [2], which includes a lot of data throughout portals throughout the US.
This is a great starting guide. As someone that heavily researched procurements for awhile these are good off the shelf resources. It takes awhile to understand the specific language and codes (NAICS, labor codes, etc) used for each of these contracts, but after awhile they start to make some sense.
I would also recommend taking a look at budget planning documents and other resources from the DoD Comptroller (https://comptroller.defense.gov/budget.aspx) continue to follow-up on task orders that are available under massive "ID/IQ" or BPA contracts in the billions of dollars (these are often withheld from the public but available to the firms that won the original work, though the actual values can be found sometimes in FPDS under sub-codes.
Additionally, many of these folks hold various BPA / GSA Schedule contracts and sometimes their lead officers will respond to inquiries.
Example: GSA Schedule 70, MOBIS, Navy SEAPORT-E, etc.
...i havent made a lot of FOIA requests lately, but there was a time when redaction would vary depending on the reason stated for the request, thus enough different requests for different reasons could be use to assemble a nearly complete document...
I seriously recommend that everyone submit a FOIA request for information that they're interested in and make it public. Either locally or federally. You'll learn a lot about the interface between you and your government.
Something to think about - Illinois FOIA allows the submission of code within a FOIA request. For example, as long as you know the schema of a database, you can submit SQL for a government owned/contracted database. The schema is often included in contract specs, but more often requires a request to figure it out. My nfp is working on some projects (through muckrock) to build out a list of schemas across all states' big cities. It's possible, but there's still a lot of work to do!