Information, please

The Bay Area leads the revolution in information sharing. So why is it still so hard to get basic public records in San Francisco?


The concept of "freedom of information" has taken on epic proportions in the Age of the Internet. Take Wikipedia founder Jimmy Wales' comments at the RSA Security Conference, which drew prominent tech luminaries to San Francisco's Moscone Center last week.

"Imagine a world in which every single person on the planet is given free access to the sum of human knowledge," Wales said in describing the project's foundational vision, his image magnified by giant projection screens. Twenty-four million volunteer-submitted entries and 490 million unique visitors later, the San Francisco-based project "has become part of the infrastructure of information," he proclaimed.

Wikipedia isn't alone in having a philosophy of open access to information. Twitter's informational updates are unbound by time or distance, while Google's stated mission is to make "the world's information ... universally accessible." For better or for worse, humans have never had so much light shed upon so many topics at once.

But even as the Bay Area plays host to a revolution in information sharing, local sunshine advocates bent on protecting the public's right to know face an uphill battle. At the state level, budget rollbacks threaten to impact agencies' ability to respond to California Public Records Act inquiries. And when it comes to illuminating the activities of government or tracing how money influences politics in San Francisco, secrecy is still the word of the day.


San Francisco city government has seemingly embraced the ethos of information sharing that's currently in vogue amid the tech world. But even as the city accelerates its commitment to "transparency" through software, traditional systems for providing access to public records are languishing.

Enforcement of the San Francisco Sunshine Ordinance, an open-records law enacted in 2000, remains anemic, and the transparency law came under attack last year with calls to scrutinize the city's cost of compliance. Meanwhile, the city agency tasked with routing political corruption — the San Francisco Ethics Commission — is still in the information dark ages.

Last fall, San Francisco Mayor Ed Lee announced he was creating the position of a Chief Data Officer as part of San Francisco's Open Data initiative, a project that displays government information online in a user-friendly format.

Powered by a Seattle company called Socrata, the Open Data platform seeks to "enhance open government, transparency, and accountability" by making data widely available. Socrata won the privilege of launching in San Francisco after snagging a Living Labs Global Award through a private nonprofit, an honor it received at an international convention in Stockholm, far from City Hall and completely outside the city's normal bidding process.

Setting the eyebrow-raising contractor selection process aside, Open Data does provide some nifty tools for information-seekers and raw fodder for software developers who can use it to configure new apps for citizen engagement. The program features data sets showing real-estate development projects, neighborhood crime incidents or campaign spending trends, among other things.

Jay Nath, the mayor's chief innovation officer, is a key driver behind Open Data. But when it comes to direct public information, he's not so open at all. Reached by phone, Nath said he needed permission from the mayor's communications office before he could answer basic questions about the system — for example, how much it cost. Due to some mysterious holdup that doesn't bode well for the city's depth of commitment to transparency, he never responded to Guardian requests for an interview. All we know is that one news report put the price at $40,000 a year.

While the information generated by Open Data is an interesting development and could prove useful in many respects, it's no substitute for sunshine.


Just got back a sunshine request from the MTA for all emails on the "Bicycle Strategy," a document that the MTA pulled out of its, er, hat.

All I got was a CDROM of enormous PDF portfolios that have scanned image versions of the emails encapsulated into PDF. This creates a new public record from an existing public record, the text email, something that the Sunshine Ordinance says the departments are not required to do.

There is no way to take these lumps of data, index them and analyze them chronologically. There is one PDF that contains 15 text emails and takes up 60mb on disk.

Posted by marcos on Mar. 06, 2013 @ 8:11 am

I can find what I need, but do not need most of it.

Posted by Guest on Mar. 06, 2013 @ 8:47 am

Here's a start

find /path -name '*.pdf' -ls -exec pdftotext {} - \; | grep "your query"

Posted by Chris Pratt on Mar. 06, 2013 @ 12:37 pm

They provided PDF portfolios of email bodies. It is impossible to select text from the PDF, although each email can be saved individually with minimal metadata.

I want the full headers of the original public documents in industry standard default MBOX format so that I can create a database that allows for indexing, searching and chronological review.

Posted by marcos on Mar. 06, 2013 @ 1:14 pm

to then cause trouble for the city?

Ever sue anyone? Did you notice how in discovery they sent you 50 packing boxes full of paper docuemtns what would take you a lifetime to go thru?

You cannot reasonably expect others to help you make your case.

Posted by Guest on Mar. 06, 2013 @ 1:29 pm

Because the law is clear on this, that the public has the right to public records in their native format.

Posted by marcos on Mar. 06, 2013 @ 2:31 pm

Jason Grant Garza here ... wow, ANOTHER ARTICLE on TRANSPARENCY, SUNSHINE and ACCOUNTABILITY ... what a WONDERFUL FARCE. What do I mean ... shall we look at the MINISTRY of SUNSHINE ( google my name to read all the articles on the CITY"S SHILL and how they are ineffective ... did the BOS CLOSE THEM DOWN for over 6 months ... were the VICTIMS to die in the NUCLEAR WINTER ? )

Yes. it is a WONDERFUL ILLUSION like JUSTICE. the RULE OF LAW and HUMANITY. Shall I speak to you about when I sunshined the Department of Public Health over who ACCREDITS them for patient/medical care and DPH could NOT provide the CORRECT information and the SUNSHINE TASK Force informed me that they COULD NOT make DPH produce something it did not HAVE? Imagine DPH does NOT know who accredits them for patient/medical care. Why, so they can be protected by illusion as was the BOY SCOUTS during the BOY SCOUTS LAWBREAKING ACTIVITY by the POLICE, COURTS, OFFICIALS, etc. Don't believe me ... look at this youtube video regarding a FALSE (CRIMINAL FRAUD) restraining order that DPH claimed in order to deny me access and accommodation, medical treatment, followup, etc on 8/15/2012 and then watch me go to the LAPDOGS (Sheriff) who tell me to go BACK to the LAWBREAKERS (DPH) and on 12/19/2012 they (DPH) break the LAW again .

Yes, if you look at the youtube videos ... you see the TREATMENT by the DA, the failing SFPD and my continuing AGAINST all odds ... just like I did when DPH broke FEDERAL LAW in 2001, had my case thrown out of FEDERAL COURT (C02-3485PJH) with TESTILYING and FRAUD in 2003 and in 2007 signed a settlement/confession with the OFFICE of INSPECTOR GENERAL admitting fault and guilt YET leaving their INNOCENT VINDICATED VICTIM for DEAD. Note it has started again ... since I still have NOT gotten an answer form either the SHERIFF nor SFPD as to who enforces and arrest for MEDICAL LAW VIOLATION as I have been demonstrating in the youtube videos. HERE is your TRANSPARENCY and NOTE the RESULTS ... sort of like the RESULTS for the four supervisors found GUILTY of OFFICIAL MISCONDUCT by the SUNSHINE TASK FORCE (Ministry of SUNSHINE) in case # 11048 for an ORDER in 2011.

Shall I show you what I got from David Chui's office regarding CIVIL GIDEON with their false and ineffective deadend lead to the SF BAR ASSOCIATION who after looking at the YOUTUBE videos told me that there was NO LAWYER who could help. Yes, the ILLUSION is GREAT, the deceit and inhumanity rival it but MOST of ALL ... the ENTIRE PROCESS is RIGGED and the INTELLIGENCE FIXED to give the appearance of correctness. Shall I tell you about the SUNSHINE to DPH in which I was denied my medial records with their false claim of HIPPA and their expert and city attorney advise that I was incorrect only to back down YET not be accountable in front of SUNSHINE ??? I have those records also inclusive of me following up with SUNSHINE to correct and being LEFT for DEAD ... why not? They (SUNSHINE) got the incorrect information ... based the incorrect decision upon it and when I tried to correct again was denied so that the "picture and framing" could be represented as help when it was the EXACT OPPOSITE and cover for DPH. This is the SHILL action by the MINISTRY of SUNSHINE to tire, vex, and annoy so that you will give up and go away.

Yes, the TRANSPARENCY proves FRAUD, DECEIT and NO incentive to change since as I stated ... what is NOT can be represented as what is and WHAT is can be represented as they ( the risk managers ) interpret and spin the TRUTH. Remember the WEAPONS of MASS DESTRUCTION that lead to WAR yet no consequence for the FRAUDSTERS ...

For more TRANSPARENCY on how the city works ... go to or go to youtube and type in Jason Garza.

Keep drinking the KOOL AID, ENJOY and KNOW you are NEXT. Here's a good example ... DPH BREAKS THE LAW with BOGUS STAYAWAY ORDER and the SFPD WILL NOT TAKE a police report/incident report in order to arrest and hold liable and accountable and THEN has the NERVE to ask me if there is SOMETHING WRONG with me ... how much more can I be transparent regarding the FAILED/RIGGED methodology?

Posted by Jason Grant Garza on Mar. 06, 2013 @ 9:00 am

Grestso ncie sposo well!

Posted by on Mar. 06, 2013 @ 2:57 pm


Posted by Guest on May. 18, 2013 @ 1:57 pm