Reminds me of a story a certain physics PhD told me about certain very known university in Poland.
One day some years ago, the university administration thought, "why are we wasting money keeping power on, running lights and aircon and God knows what, when the buildings are vacant?". They soon turned that thought into action, and when a long holiday break came, they just shut off the power to the buildings as soon as everyone left for the day. About a week later, the PhD I mentioned and his co-workers came back to the lab, saw some droplets on the glass, and almost had a heart attack.
Turns out, after a week of being without aircon to dry and warm it, the air was just about to cross the dew point threshold, and if it did, it would kill the optics on a million-dollar research laser they recently got installed.
I hear that since then, they're a little bit less pound-foolish.
Your story now reminds me of our new plant manager back in the days of the last financial crisis.
He was newly appointed as the supreme leader at this automotive plant. Uncertainty was high, this being 2007, so people were let go, and costs reduced with any means possible. So he went into production and one day decided that it is too cold in the storage area. He orders his people to increase the temperature with a few degrees since that would save tons of money. People protest, he hears only "blaa-blaa", and his request is implemented.
A few days later, it turns out that barrels upon barrels of chemicals in the storage area needed to be kept under a certain temperature. So the stuff solidified and it was now only good for the dump site. Out to garbage the chemicals went, new stuff was ordered, production was stopped until the new stuff arrived and temperature was restored back to normal after that.
The required storage temperature for chemicals being stored is a pretty obvious thing to document clearly. It's also not a wild idea to have adjustments to the storage conditions follow some sort of review process.
Chesterton's fence is sort of about understanding something that isn't necessarily obvious, changing storage conditions in an industrial environment without checking requirements is neglect rather than ignorance.
I suppose I am arguing that this is far more obvious than 'reasonably avoidable ignorance'.
I suppose I am also arguing that there is a mechanistic requirement of the role 'plant manager', where they shouldn't make changes without understanding the impacts. That's somewhat different than ignorance, because it's pretty much their job to do what it takes to act correctly, avoiding assumption and the like.
I genuinely like your argument: although I've seen a few too many managers that exemplified the Peter Principle it's not unreasonable to hope that a "plant manager" is aware that a web of process exists, even if they can't immediately cite it.
To be fair, this particular power outage in California isn't to save money. It's to prevent forest fires. Also, how important can these experiments be (in CA) if they don't have sufficient backup power at a university??
Edit: If a Uni has a multi-million dollar laser, it should have a few thousand dollar backup generator!
No, it's to prevent forest fires, which is why the policy was blessed by the federal judge overseeing their felony sentence and probation for the crimes they were convicted of in the 2010 San Bruno gas explosion (who is involved in the fire safety issue because PG&E’s role in last year's fires was charged as a parole violation), even though he noted that the reason it was necessary is because PG&E underfunded safety maintenance while issuing massive dividends.
What exactly is the usefulness of a felony as applied to a corporation?
As applied to an individual, it is capable of ruining your life through privation of constitutional rights (firearm ownership, right to vote, permanent mark on background checks make it increasingly unlikely you'll ever hold any sensitive position, etc...)
What does it mean for a corporation to be a felon? It sure seems like the concept is discontinuous in terms of scale. Obviously, there's an extra layer of court oversight that's been mandated, so some autonomy is lost. Still doesn't seem though like the the punishment for a corporation is concommittant with the general treatment of an individual having committed the same relative severity of infraction.
In fact, I'm surprised that anyone even wants to do business with anyone involved with the company during the period the felony was handed dowm.
The more I think about it, it seems like all the "felon" label does is act as a signpost for discrimination being okay.
Anyway. Brain dropping. My apologies for the tangent.
They have backup power which can get them through any expected power outage. They don't have unlimited backup power that would be able to get them through a "apocalypse" or major terrorism scenario where the world is ending and power will be out for more than a week. Preparing to survive an apocalypse or major terrorism incident is possible. Build all facilities a mile underground with a self enclosed ecosystem. Might cost trillions, but can be done.
PG&E has intentionally deployed, at short and inadequate notice, what is equivalent to a terrorist attack on west coast infrastructure. No one prepared for or expected that scenario. Past events have shown that PG&E management are sociopaths who intentionally engaged in mass criminal negligence which resulted in dozens of deaths and billions of dollars of property damage. Even under the oversight of a captured court, they continue with their games.
If there is any sanity left in California, the USDOJ would bring the executives who deployed this plan and the judge who signed off on it under terrorism charges. If any deaths occur as a result of these antics, it should be capital charges.
You almost lost me at the apocalypse metaphor but I catch your drift. Weeks at a time of no power aren't easy to be prepared for, but I feel like this event should be one more reason we need more distributed power generation in the US. Less single points of failure.
Yeah sorry I didn't mean to suggest a religious apocalypse, but any civilization-ending event which apocalypse also means: whether cooling and global glaciation, warming and ocean inundation, all-out global nuclear exchange between superpowers, the eruption of the Yellowstone caldera, large meteor impact, "inevitable" Carrington event, 1 in 1000 year worldwide tsunami, etc. But it can be local too and without outside support is the same as world ending if you can't get out. If you have power loss for weeks in an area that is totally power dependent you start to see civilization meltdown. The situation in Puerto Rico recently was an example, another is the Haiti earthquake before that. Neither has recovered years later.
You make a good point about distributed distribution. Would have to be very local to avoid this wildfire problem as PG&E is doing (shut off power instead of deal with actual issue). The wildfire problem is one that is decades in the making and coupled with poor open areas management by government policy in general and not just the power company, in particular not allowing periodic fires to burn out underbrush. I had this huge pile of brush in my backyard. When it burned it was very dangerous. Now I make sure to do more regular smaller burns as giant burns are not easily manageable.
I like the solution of small Hitachi style safe nuclear reactors that service neighborhoods of around 2000-10000 people, can't meltdown or breech, are useless to terrorists, and require no maintenance. Doubt we'll go that way though.
> if they don't have sufficient backup power at a university??
This. If you've got a multi-million dollar experiment, gee I dunno make sure you're in a building with a generator?
"hey Mr. Government, I know you gave us that multi-million dollar DARPA reserach grant, but uh, someone hit a telephone pole drink driving on the weekend and, uh, power was out for several hours and uh yeah, that 2 years of research, uh, we gotta start all over can we have some more funds please?"
I mean..it's a bit of A and B. From the outside (i.e. from a non-CA resident), PG&E appears pretty incompetent and/or insolvent and can't properly manage it's infrastructure. Is their main driver still really about saving money?
note: it's not as simple as that (part of the fire problem is due to deferred maintainence; AKA we could have spent money 10 years ago to address this).
However when you have a laser system it's not as simple as having a backup generator. When the entire building power is affected, the AC system stops working, and all your nicely aligned optics become unaligned. If you fix them at that time, then when the AC restores, you have to realign them.
There are hundreds of externalities that affect productive state of the art research.
Reasoning is a tool we as humans use to justify our beliefs and actions (and not an internal process which shapes belief as often depicted). Communication is naturally enviable to help strengthen that reasoning. Communication <-> reasoning -> belief+action is not natural and should not be the expected process from one's future self or other from humans.
My university is not a top tier research university like UC Berkeley by any means, but our co-generation plant is able to power the whole campus in the case of a power cut. Even when the central plant gets knocked out (a squirrel did that last year), individual buildings have backup generators. The building where I work has three of them and all of our -80s are on back up power or could be quickly shifted over. That is several layers of redundancy. We have done a lot of planning for events like this. I'm a little surprised that a more research focused (and less resource starved) institution like Berkeley wasn't more prepared.
It depends a lot on individual departments. Most astro groups I've been in have had very competent sysadmins who aggressively lobby for fault tolerance and assume that the university proper is incompetent (the very best kind of angry greybeards). So all the server rooms have enough UPS capacity to gracefully kill servers, data is properly backed up and so on.
At some places like MSSL we even had an onsite diesel generator becuase there were flight critical experiments going on all the time.
Many other faculties don't need that level of resilience so they don't bother. I'm not sure what the situation is for wider engineering or bio/chem. I imagine there are serious health and safety concerns for things like fume hoods that must be able to evacuate gases in the event of a power failure.
Berkely has a few generators though so I think these particular complaints are due to cost tradeoff decisions made decades ago and a general lack of foresight to predict the utility power completely shutting down.
Oftentimes, multiple failures cascade. About 2 decades back at Ohio State University, there was some renovation work going on. Due to this some sensors were switched off. This was wintertime and heating was on. During a weekend something failed and temperatures in a few rooms went as high as 50C. These rooms were used to house mice and rats for several long running experiments and all of them died or had to be euthanized as the data was unusable. Some people lost years of work.
>This map lists all unclassified Cyber Squirrel Operations that have been released to the public that we have been able to confirm. There are many more executed ops than displayed on this map however, those ops remain classified.
>Confirmation for all ops has been preserved by the Internet Archive's WayBack Machine whenever possible.
>"I don't think paralysis [of the electrical grid] is more likely by cyberattack than by natural disaster. And frankly the number-one threat experienced to date by the US electrical grid is squirrels." - John C. Inglis, Former Deputy Director, National Security Agency 2015.07.09
My theory is that all the not-resource-starved universities are suffering from administrative bloat, while universities that stay lean have the experts make these kinds of decisions out of necessity.
I did a tour of the power plant at Notre Dame university (in Indiana) back when I was in high school. They have their own coal plant on site where they get a portion of their power along with diesel generators as backup for that. Apparently they get the rest of their power at a reduced rate from the local utility because of an agreement that they can temporarily go fully off grid if the power company requires it.
I'm glad I live in the city of Santa Clara, where I am served by Silicon Valley Power, one of few exceptions to the PG&E monopoly in northern California.
Silicon Valley Power serves many homes in addition to large companies including Intel and Nvidia, and they do so with lower prices and better quality of service than PG&E. I will be very unhappy going back to PG&E if I move to another city.
Electrical utilities don't need to be a government-granted monopoly like PG&E, which combines the worst aspects of government-run and private-sector business. Silicon Valley Power proves there are better ways to run electrical utilities.
I was born and raised in the Bay Area and remember when there was a vote that would have brought SMUD in instead of PG&E. I don’t ever remember PG&E coming out to our property (South Bay), despite having large trees near power lines.
I’ve since left the Bay Area and now have SMUD. They come out every year to check brush and trees. This past summer they cut down a dying tree on the property at no cost. The arborist they sent out was extremely knowledgeable and didn’t mind that I took his time to ask about native replacements, etc.
> Electrical utilities don't need to be a government-granted monopoly like PG&E, which combines the worst aspects of government-run and private-sector business. Silicon Valley Power proves there are better ways to run electrical utilities.
Is Silicon Valley Power not a government-granted monopoly? How does their business model differ?
Silicon Valley Power is a nonprofit operated by the city government. It's a little island in the middle of the sea of the PG&E monopoly. I wouldn't consider it a monopoly in comparison to the massive scale of PG&E, even though it is operated by a city government.
Statewide utilities are bloated and inefficient, and they abuse their monopoly power. PG&E acts like the only choices are to cause fires or turn off power when it gets windy -- it faces no pressure to find a better solution because there is no competition.
Small-to-medium municipal utilities that serve smaller areas can serve their customers much better, as Silicon Valley Power demonstrates -- if people don't like it they can move to a neighboring city. In contrast, PG&E is almost impossible to avoid.
California should give more cities and counties the ability to experiment with this model. Competition between municipal utilities is better than granting a single company a monopoly on a utility millions of people depend on.
Here is the important line for you to chew on in wikipedia article of SVP..
<wikipedia>
SVP receives generation produced outside Santa Clara via transmission facilities owned and operated by PG&E under the direction of the California Independent System Operator (CAISO).
</wikipedia>
About 2/3rd of SVP power comes from outside. If PG&E cuts certain transmission lines the good folks of Santa Clara will feel the bite, a bit lesser perhaps, but a brown-out can be willfully imposed by PG&E
Utilities with regulated monopolies need to be on a tight leash from a regulatory perspective. They trade operational freedom for a reliable return on assets.
This changed in the 80s and 90s with deregulation. Some of them even turned into growth stocks or did other financial fuckery that made investors money at the cost of rotting out the business.
End of the day, local municipal utilities or co-ops almost always deliver a better level of service at a lower cost. Smaller scale makes field service more efficient and reduces overheads like executive compensation. One municipal utility I’m familiar with has a civil service scale for all staff — the head guy makes $140k.
i think what the parent poster might have meant was that for-profit state monopolies are bloated and inefficient (as opposed to non-profit public utilities)
that was my interpretation (from the context) anyways... maybe i’m wrong ^^;
The threshold for keeping your job is very low when there is no competition.
Compare how cops act (who are damn near impossible to fire and have a local monopoly) with how private security acts (who can be fired on a moments notice and who's customers are not locked to a particular provider).
Let us clear up the terms we're using: They are both monopolies. But PG&E is investor owned, publicly traded. Their incentive is money, followed by quality of service.
A government entity in terms of public services tends to have "getting their job done right" at the top of the list, albeit at a lower cost when possible.
> A government entity in terms of public services tends to have "getting their job done right" at the top of the list, albeit at a lower cost when possible.
not my experience. Most government entities try to raise as much money as possible to protect their jobs and raise their salaries. gettingthe job done right is not remotely near top of the list. See SFMTA or MTA (new your public transportaion) as 2 examples. Or watch IKIRU to see the same attitude in Japan. Yes it's a movie but it rings true to people's experience that goverment never tries to "getting their job done right"
What exactly is "your experience"? You've ridden two subway systems and attributed their failures to not using your pet economic structure, and watched a fictional movie from 1952?
It doesn't take a whole lot of googling to find thousands of example of poorly run government services. I gave 2 I thought were obvious examples but you can search and find 1000s more.
I'm not defending evil and corrupt PG&E. I'm only responding to the idea that government services are magically incentivized at "getting their job done right". Their incentives rarely align to "getting their job done right".
Other examples off the top of my head, public schools in the USA who have more money per student than pretty much every other country but have worse outcomes. I don't have a solution but clearly they are not "getting their job done right"
Another example might be the USA's militarized police force killing its own citizens. Some people have arguied they're incentivized to buy all the military equipment because it raises their budgets. Again, they are not targetting "getting their job done right". And again I don't know the solution. Just pointing out making something a government serivce does not magically remove all bad incentives.
> It doesn't take a whole lot of googling to find thousands of example of poorly run government services.
Perhaps, but you didn't say "Some government services are poorly run". You said:
"not my experience. Most government entities try to raise as much money as possible to protect their jobs and raise their salaries. gettingthe job done right is not remotely near top of the list."
You have not yet given an example of this. You're just pointing to random organizations' failings and saying that they're caused by incentives specific to government, without any evidence whatsoever.
Are MTA and SFMTA's failings because they are "trying to protect their jobs" or because they have to maintain aging infrastructures in a culture that idolizes cars?
Are public schools' failings because they are "raising as much money as possible" to "raise their salaries", or are they because of mandatory religion-based curriculum, mandatory standardized testing determining funding, lack of healthcare/childcare/nutrition support for parents, and parents being incarcerated?
It's clever of you to bring up police, because you can look through my previous comments and see my political bent, but keep in mind that the discussion at hand is whether government or corporate incentives lead to better results. Looking at where the money goes when overly-militarized police purchase military equipment, I think it's hard to attribute this to government incentives as compared to corporate incentives. You can't point to corporate meddling in government causing problems, and use that as justification to put the corporation in charge.
Saying "not my experience" at the beginning is pretty disingenuous, because it's become increasingly clear that you don't have any relevant experience.
The Japanese train system is privatized. In fact it's often held up as one of the most successful examples of taking a service from government run to privately run and having it wildly succeed.
The difference is government entities can be made democratically accountable which means effective pressure can be brought to bear on them.
It doesn't mean they are setup that way or that they (or the electorate) don't become captured by vested interests but it is the most effective way of keeping natural monopolies in our economy free from corruption (heavy regulation can also work).
>The difference is government entities can be made democratically accountable which means effective pressure can be brought to bear on them.
Or they can be shielded from accountability because government bureaucracy. Imagine if the MTA were a private entity. Their contract would not be getting renewed (to put it mildly).
On the contrary, MTA would definitely be renewed if they were a private entity--there isn't a competing New York transit company who has built out a complete rail system underneath the streets of New York just waiting to be used.
Assuming there are alternatives. I still see IBM/Oracle/MS/Boeing/Lockheed and tons of others still getting lots of business from customers they have burned.
The point also ignores the power of network effects -- certain industries are prone to monopoly. It is only natural that government oversight of these natural monopolies follows.
The government made it clear that PG&E will be harshly punished for operating in high fire danger conditions, and so PG&E is not operating in high fire danger conditions. This outcome is exactly how government oversight is supposed to work.
They're in bankruptcy proceedings, no? Hiring more staff in that situation might be difficult.
This is really a situation where bad incentives were created (because as far as I can tell the California state government is pretty bad at doing good incentives in general) and then people are surprised when they lead to bad behavior...
The problem is, the government always steps in and grants a monopoly before a natural monopoly develops by itself, thereby creating a self-fulfilling prophecy.
Maybe a natural monopoly would not have developed statewide. Maybe the government makes a bad situation worse by picking the wrong winner before the market decides. Maybe the government should allow competition if cities and/or private businesses are willing to give it a shot -- it works pretty well where I live.
I agree that municipalities should be given latitude to attempt to replace statewide utilities. But they should have some quality standards -- you would not want a rinky dink WISP with an invisible privacy record managing the internet access for schoolchildren.
Even taking it as a given that last-mile power distribution tends towards a natural monopoly for every neighborhood, this still doesn't mean that this monopoly needs to be agglomerated across such a wide area.
This overt dereliction can only come from an entity that is "too big too fail". If this were one or two cities' local power companies, it would not be national news. Residents of those communities would be showing up at city meetings going "wtf", and could easily point at adjacent communities not doing such things as a reference.
PG&E has been doing blackouts and brownouts more than basically any other electric company I’ve ever experienced anywhere in the world since I was a little kid. These blackouts may be bigger, but if you’re an institution the size of a university, you should have significant disaster preparedness infrastructure. I mean US universities run police forces! I’m sure they could find a way of designing redundant power into their systems. They might even save money in the long run, and wouldn’t need to worry about campus hospitals or public safety going down in an outage (I’m sure some of that is already taken care of, but why not go all the way).
Many universities have "cogeneration" plants that produce electricity, steam for heating, and also hot water. However, they usually don't generate enough electricity to power the whole campus, so the administration chooses who gets it and who doesn't. If these sorts of outages are going to become more common, then the management is going to have to re-evaluate the cost of providing backup power for all labs vs the cost of lost research.
I recall some time back, the power generated with cogeneration needed main power with the right frequency to condition(?) it. The cogenerator couldn’t run during a blackout.
For a blackout, the diesel generators kicked in (and could completely power the platform).
Power plants need to be connected to other sources so their AC frequencies can be in phase. If the multiple AC waves were out of phase, you’d cause all sorts of issues. It wouldn’t surprise me if the cogen plant was designed to only work in concert with external power, but not in place of it.
Emergency power would be expected to work independently of mains power.
True for inverter based generation (most renewables like wind and solar), but generally not for rotating machines like most Co-gen plants. Some rotating machines (steam turbines generally) won’t have governors for frequency control, but usually in a CHP plant they would have a linked combustion turbine that does. Rotating machines that are paralleled with the rest of the grid are by default in sync with all of the other rotating machines if there is not some sort of fault situation at the plant or on the grid. If you try to parallel a generator with the grid that is out of sync (frequency or phase), the grid will always win and pull the generator into sync (potentially destructively for the generator).
The most likely limitation of a typical power plant operating disconnected from the grid would be if it could “black start” or not. Cogen plants would normally have this capability. Essentially the problem is whether you have enough power available from backup generation to bring all of the plant auxiliary systems online and get the generators initially started and producing power. Once started you set the governor to maintain 60.0Hz (if in North America) and begin slowly bringing on load so you can ramp the generation to match it.
The easiest solution to a future blackout would be to match the load in your island area to the output of your plant and then disconnect from the rest of the grid before or at the time the blackout occurs. This way you have avoided the black start situation completely. The blackout detection, disconnection, and load balancing steps can be automated to occur nearly instantly if the system is designed well.
The main issue for Berkeley is that the cogeneration plant doesn’t seem to be sized to accommodate their islanded load. This is especially problematic if you lose a generator while islanded because then you will have instant frequency collapse if you were already on the edge of your plant’s capabilities.
Wow, that's annoying, too bad that they wouldn't have a way to disconnect. IIRC, Princeton was able to operate some buildings off of the cogen plant during Hurricane Sandy, so I guess they had things engineered differently.
I don’t know — just imagine how many places the mains power is tied into all of the buildings. Each of those places would also need a mains cutoff, and they would need to all be thrown/active at the same time before the cogen plant could be activated independently. At a place the size of UC Berkeley, that would be a pretty big cost (and logistical nightmare).
If the system wasn’t designed for the switchover from the beginning, I can see how it would be considered impossible/not practical to do. Especially when this type of “blackout” was thought to be rare.
I’m sure the prior assumptions would be reconsidered today.
> just imagine how many places the mains power is tied into all of the buildings
Not many? As in, I don't think many buildings would have two separate power feeds. I'd expect a small number of very large connections from the cogen plant and the outside word into local distribution network(s), and then everything is single-path from there on.
But isn't that just because of legislation? As far as I can tell the system could just as easily throw a switch, test to make sure the switch was thrown, and then safely power locally. I know in California this is law, but I have never heard of a very good reason why it is technically difficult or inherently more dangerous to "air gap" your local power at the building rather than at the battery/power source.
Agreed. When I lived in Wisconsin the power would go out frequently during storms. Immediately I'd hear some neighbor's natural gas generators kick in and then a few minutes later I'd hear a few neighbors fire up their gasoline generators.
In both cases, they would flip over their home (or a sub panel) to the alternate power source (and disconnect it from the upstream grid). I looked into these systems and the natural gas ones could do this automatically using a kit available at Home Depot, and the gas generators required a manual switch, but were still very simple.
I looked into Solar in California and when I found out that most people can't use their systems during blackouts I asked about these generator systems that switch over and they basically said they had not seen equipment that could it and they didn't believe it was allowed anyway.
Solar is different from a generator, in that a generator is already producing AC, and as long as it produces a reasonable frequency and waveform you're good. Solar produces DC, which you then need to convert to AC, and the inverters that are typically installed are not able to do their own waveform shaping, depending on the grid to do it for them. It's possible to get inverters that handle this correctly, but they're more expensive, so typical home solar installs don't use them.
Or is it a horse and cart problem? Since there is legislation against it, there is not a demand, and prices have stayed high to sell them as "off grid" tech only? Most anybody would spend $500 extra on a 30k system for guaranteed power during blackouts.
There is no legislation against it where I am (Massachusetts), and there is still very limited demand for it because of the pricing.
> Most anybody would spend $500 extra on a 30k system for guaranteed power during blackouts.
To be clear, to get "guaranteed power during blackouts" out of solar, you need both a more expensive inverter ($1k or so last I checked) and batteries ($5-15k last I checked). If you're OK power only when it's sunny _and_ a blackoutm you might be able to do just the more expensive inverter, maybe. I'm not actually sure whether that work without a battery sink for your unused current.
Any inverter is expensive. I am currently pricing out a small system and just found out today the battery ready inverters don't cost more. In fact I am going without batteries due to a niche use case, but the inverter will be battery ready just because it (SolarEdge) is the best deal overall for the system. You are right about battery cost, but just having one that can get you (frugally) through the night would be a world different than just being shafted.
As to battery requirement, I can't see how attaching a wire to a panel would cause it to burn up. The wire would charge and then it would be the same thing as if the panel was disconnected with the same amount of latent charge in the panel.
Nothing, since they are disconnected, so there's a voltage across the gap but not enough to spark. And you don't care precisely what the voltage is, because there's nothing happening with it.
But if you don't disconnect them, now you are maintaining a voltage across whatever your load is, but that voltage depends, for solar panels, on the current being pulled from them, as I understand. And you need to maintain the voltage in some nominal range. Maybe it all works out, but I'd have to do a lot of digging to be sure...
Inverters are only used for DC to AC conversion, cogeneration sets are conventional prime mover setups that output 3-phase AC directly. The problem likely has to do with islanded grid stability, without the grid you have to take care of power factor correction and grid voltage regulation yourself, this can get pretty dicey if you don't have direct control or knowledge of the type of loads you'll have.
In an islanded mode of operation you also need to match load and generation to maintain system frequency, which is tough to do with cogen and non-controlled loads.
Basically, it can work, but the whole system is going to need to be engineered for it from the outset.
I work for a significant tech company in the Valley and we are in the PG&E grid, but we have diesel backup generators that we immediately fired up when power was cut to a few of our buildings. “Tied” to the grid doesn’t preclude backup power. As long as we can get fuel trucks to our buildings, we can run indefinitely.
If you think that lack of disaster preparedness is bad, apparently many dialysis clinics in California don't have the generation capacity to continue operating during a power outage like this one. No big deal, I mean they're only keeping people alive after all...
Seemed like it should be fairly common. Saw in a different comment that maybe there is some issue with being able to switch over to only being powered by that plant.
Seems odd though, since one of the things I thought was always touted about our municipal power plant growing up was that they could power the town if external power got cut off (But didn't normally because our generators were more expensive to run)
Not really: I was recently in Juticalpa, Honduras (it's a major city, like SF) and we did not have light for three days. Similar experience in Rio Dulce, Guatemala. In the third world, doing science is very difficult, because the infrastructure is just not there.
But the cost of providing full backup for 5 days has got to be pretty significant. Likely it means diesel generators the size of locomotives, sourcing fuel in what may also be a fuel shortage crunch, maintaining and testing this plant.
EDIT: I screwed this all up and it's wrong. I tried. I'm sorry. Leaving it up so someone else can give it a go. And there's already a bunch of comments.
Washington University uses about 376,394 kWh/year. [1] That's about 7,841.54 kWh/week. Washington pays about 13.1 cents per kWh for power [2]. Running on the electric grid alone this would cost $598.59 per week. This isn't what they pay though, because they offset much of this usage with renewable energy like solar panels.
That would take a 50kw generator to power. Assuming they don't power all systems during a blackout they could probably get away with a 25kw generator. For argument, let's assume a 50kw generator full load would probably use about 4.0 gallons of diesel fuel per hour. [3]
At $3.64/gallon for diesel [4], at 4.0gallons/day, for one week the university's fuel cost would be $2,446.08.
To recap, the university would spend $2,446.08 to generate electricity they could purchase for $598.59 from the electric company during a 7 day blackout at full capacity.
Yeah, from the link, the quoted number is actually the amount Wash U saved by changing light bulbs.
> the school has completely replaced their lights with more energy efficient, low wattage bulbs that save an average of 376,394 kW hours per year in energy
The 376,394 kWh/year is the savings they got from going to low wattage bulbs.
"Throughout the rest of the campus, the school has completely replaced their lights with more energy efficient, low wattage bulbs that save an average of 376,394 kW hours per year in energy" [1]
And your prices are from the Washington-Arlington-Alexandria municipal area, not the St. Louis area in which Washington University St. Louis is located.
As user jumpingmice pointed out the obvious error in your numbers, the figure you've cited from [1] as the yearly power used was the yearly power saved by adopting efficient light bulbs:
> Throughout the rest of the campus, the school has completely replaced their lights with more energy efficient, low wattage bulbs that save an average of 376,394 kW hours per year in energy. Their overall lighting plan has saved more than 20.6 million kilowatt hours in total.
I’m guessing the problem is in the capital cost of having a generator system installed. Getting procurement, university real estate/capital projects, a general contractor, and any deans/admin involved to build the system.
I have just assumed that all science buildings had backup generators by default. Movies may have tricked me. I think all hospitals are required to though, which makes sense why the researcher moved his metabolic stuff to UCSF
Here in France I pay 0.138c (€, thus 0.15c in $) for 100% renewable elec. That's like twice as much in CA... for what I guess is not environment-friendly by a long shot?
I'd really have thought with the USA fracking their soil like it's apocalypse tomorrow, and pulling out of any carbon commitment on top of that, at the very least it would yield considerable short-term domestic advantages, i.e. cheap elec. I'd have thought you guys paid like 5 or 10c kWh. Next thing you're gonna tell me you pay more than $1.2/L ($4.6/gal) of diesel fuel.
Actually just checked, diesel fuel is $3/gal, regular petrol slightly less in general (~$2.5) but more in CA.
There are parts of the US where electricity is cheap, and a few parts of California where unleaded premium fuel is at or near $4.99 per gallon. The pricing doesn't make much sense to me.
California has a different gasoline mix with the seasons for emissions reasons. And every time we switch between winter and summer mix, the few refineries here always seem to have mysterious maintenance emergencies. Not sure why they always need to shut down for a few days right then, but it sure makes for a supply crunch and a price bump.
Surprinsingly enough, the French nuclear electricity costs about as much as hydrocarbons. It seems the "almost free electricity!" slogan of the 1960's was less than accurate in its predictions.
Lots of valid reasons though, I don't know that we've been particularly bad at this. Nuclear fission is just not the insanely cheap producer economic theory would suggest on paper, at least not in all configs/countries.
CA has much higher electricity rates than most other parts of the US. For example, in Texas electricity costs about $0.11/kWh; Washington it costs about $0.09/kWh; and the average in the US is ~$0.13/kWh.
Probably the disparity is in part because of how likely it is in CA for power lines to burn down entire towns, thanks to a dry climate with many large forests adapted for fires; CA is so naturally fire-prone that some native Californian tree species, like redwoods, require large forest fires as part of their reproductive cycle. So more line maintenance is required (although PG&E has probably not done enough...).
Redwoods do not require forest fires for reproduction, and no native species require large forest fires.
There are plenty of native plants that need fire to reproduce, but they do better with smaller fires that produce heat to open seeds but don't damage topsoils.
The large fires we have now are the result of climate change, long-term fire suppression, and unsustainable forest management. It's not normal for California, even though there are fire-dependent habitats here.
I agree that CA is much more at risk of fire now than it naturally is, thanks to short-sighted human endeavor; I was only meaning to point out that it isn't correct to compare CA electrical rates to France as a measure of U.S. vs French rates, since CA's are so much higher than the U.S. average due to fire risk.
In New Zealand I pay ~29c US/kWh for 80% renewable generation during the day. 11pm-7am it runs about US10c/kWh. Handy for charging the car. I live ~500km from the main generation source as well, until a larger local geothermal plant comes online.
It's fascinating how these PG&E threads are full of recriminations, but not solutions. The power went out for 2.6 million people (on 800k accounts). No natural disaster or unanticipated software bug caused this problem. The situation is purely a human coordination failure. The state is doing nothing except issuing sternly worded statements when it could be changing policy. This odd mood of angry, passive resignation is disturbing.
This whole situation represents a loss of "social technology". Just as material technology is a bag of techniques for organizing matter into useful configurations, social technology is a bag of techniques for organizing people for a useful purpose. When a society loses a technology, be it material or social, it loses a capability. It becomes unable to do what used to come easily to it. For 120 years, California was able to keep the lights on. Now it can't. How is anyone supposed to not see this situation as a kind of foreboding regression?
This new power grid unreliability is far from the only example of our struggling to do something that came easily to us 20, 30, or 40 years ago. We've definitely lost something, although it's hard to pin down exactly what.
I don't understand this comment. The grid is as reliable as it's ever been, but PG&E has been strongly incentivized to prevent wildfires. They're reacting to that incentive by preemptively shutting down the grid. There's no way of seeing the future where they didn't do that and whole cities burned to the ground, but based on recent history it's very likely that that's the future we're avoiding.
Fun fact: Two days before to the camp fire PG&E sent notices to customers that they may turn off power due to a forecast of high winds and low humidity as indicated by a NWS red flag warning. PG&E decided not to do so. The weather came, as predicted, and contributed to the fire. Shutting down the power like it was announced would likely have prevented it.
I'm not sure if public outcry was a factor in their decision to not follow through with the shutdown before, but you can bet that it isn't going to be a factor now.
Sure. I accept that there's a hugely elevated risk of fire right now. Maybe turning off the grid was a prudent move. But the reason that fire risk is high right now is that for one reason or another, the grid hasn't been properly maintained, and that's not because we forgot how to build chainsaws.
A society is a complex machine. There are thousands of interacting incentive structures, contractual obligations, and soft expectations that combine to allow us to accomplish more than we can do alone. For a long time, this machine operated smoothly enough to provide the power grid with the maintenance it needed to operate safely in 10MPH winds. The social machine broke down, so the maintenance stopped, and because the maintenance stopped, the grid became unsafe to operate.
Why did the social machine break down? Why did nobody fix it? Why are we having so much trouble operating this machine smoothly when our ancestors were able to keep it going for over a century?
It's not clear to me that these outages are due to maintenance issues rather than due to an extremely conservative risk posture because it looks like PG&E will be held accountable for grid-started fires even where maintenance wasn't an issue. E.g. PG&E's current filing for bankruptcy protection was triggered by the Tubb's fire, which Cal Fire's investigation concluded wasn't even caused by PG&E.
If anything the social machine that appears to be broken is how we're apportioning liability. Electrical grids, even well maintained ones, sometimes throw a bit of sparks or start small fires. No amount of tree trimming is going to prevent fires if strong winds blow dry flammable brush across high tension lines.
You can find zillions of videos on the internet of power lines going down and sparking small fires... but in California if a line goes down in the wrong place and in the wrong weather it causes tens of billions of dollars in damages.
It's not clear to me that its economically possible operate an electrical utility under the risk that the smallest spark could result in being responsible for billions in damages.
The reliability of the grid is a combination of the technical reliability of the grid and the social ability to reliably coordinate the resources needed to operate the grid. You need both systems to work in order for electricity to flow. As you point out, there's nothing wrong with the grid on a technical level. But the social element failed, so electricity stopped flowing. When the electricity stops,
it doesn't really matter whether the power is off because of a technical or a human problem. The lights go off either way.
The wind that shut grid blew at 10MPH. (Granted, it was forecast at up to 45MPH.) Did we not see winds of this strength during the 120 years* that we didn't shut down the grid over human coordination failures? If PG&E had to shut down the grid to avoid fire in light to moderate winds, it was because the people operating the grid were unable to coordinate the resources needed to operate safely under these conditions. What we've seen here is a breakdown of human systems that used to work reliably. We used to be able to operate an electricity grid reliably in moderately windy conditions. Now we can't. That should worry everyone.
* Yes, I know about Enron. That we've had two grid failures in 20 years due to human-coordination breakdown is a huge red flag.
Edit: phrasing, note that the winds were forecast to be stronger than what we ended up seeing
Where are you getting this 10mph number? There's a wind advisory in effect with gusts peaking between 35-45mph this morning[1].
> NWS forecasts call for winds to be slightly weaker Thursday throughout the Sacramento Valley than they were Wednesday, but a wind advisory remains in effect through 6 p.m. as gusts between 40 and 55 mph remain possible.
I was reading https://www.cnbc.com/2019/10/09/pge-cuts-power-to-millions-i..., particularly this bit: "
Wara said a forecast Tuesday night showed winds in Mt. Tamalpais, a nearby mountain, were less than 10 mph (16 kph). “There is zero wind where I live,” Wara said. “PG&E is required to shut down based on observed conditions, not forecast conditions.”"
The article goes on to mention forecasts of 35-45mph gusts. I'll edit my original post. My point stands: whether we're talking about 10mph winds or 45mph winds, we're talking about weather within the normal range of variation and conditions that electricity grids elsewhere handle admirably.
> within the normal range of variation and conditions that electricity grids elsewhere handle admirably.
California has very different conditions than most other places in the US. It gets no rain for 9 months a year followed by a wet winter which results in a lot of vegetation growth. And that pattern is getting more extreme. You can't compare that to a wetter place, or even to a much dryer place with little vegetation.
> Did we not see 10MPH winds during the 120 years* that we didn't shut down the grid over human coordination failures?
How can you be so sure that many major fires in PGE territory over the last 120 years weren't caused by utility equipment? And why do you assume that conditions haven't changed over the last 120 years in a way that makes fires more likely or the damage to humans more extensive and expensive?
> This odd mood of angry, passive resignation is disturbing.
This seems to be the hindrance for most progressive changes in California. Housing is quite similar. I almost feel like some people are intentionally making the situation worse so as to provoke a far more radical change rather than allowing problems to be solved.
I'm finding it hard to put this idea into words in relation to a kind of 'civilization-scale', but the analogy is the disappearance of the 'killer instinct' after a certain time dominating some field of competition. Maybe not even that, just the blind spot of being the one sitting on top of the mountain.
Looking at how the winners are doing things in order to best them. Once you are the winner, the whole methodology flips on its head at the point of realization of 'we're now the best'. There's an amount of legacy baggage that comes along with it that's very hard to dispose of.
X got us to the top, so let's keep doing X forever since we've got our whole structure aligned to the methodology behind X. Inevitably, the things that made X the right answer at the time will slowly change, and that change won't be noticed, or accepted, until it's too late, and the entity doing Y ends up besting the entities doing X.
I'd say 70-90 years is starting to look like the 'world superpower' transition period.
Once infrastructure starts failing (bridges, apartment buildings, power grid) in ways that would get politicians fired and the responsible business directors given jail-time a decade or two ago, and yet none of these punishments is even considered likely in the current state of the world, then we're getting to the end-game. It's a failure on so many levels. It's the spiraling failure of 'X'.
> We've definitely lost something, although it's hard to pin down exactly what.
The notion of collective good. The government may not have been drowned in the bathtub yet, but the idea that there's any higher purpose to humanity than maximized profit extraction for the rentier class appears to have won.
This is such a screw up. Personally I think this is PG&E trying to negotiate with the State to have it indemnify them against wildfires so that they neither have to pay to maintain their infrastructure, nor pay out large damage claims when that infrastructure burns down a town or two.
I really wish the PUC wasn't captured, otherwise we might be able to fix this problem.
100 years of US govt forest policy provided the fuel, PG&E provided the sparks.
Edit: this is not an anti-government rant. There is far more vegetation in the wildlands of California than is natural due to forest fire-fighting policy dating back 100 years.
Probably that many places have pursued policies of preventing forest fires entirely when they are actually a natural part of ecological development. If you keep preventing forest fires, you end up with a lot more potential fuel buildup than if you let them burn once in a while.
I'm sure someone will point out that CA does controlled burns on occasion, but it's not the same thing.
Basically controlled burns are the key to preventing a catastrophic wildfire but the US Forest Service and its Smokey the Bear slogan said "fire bad" which caused vegetation to grow uncontrollably over the years. Combine this with the high winds, extremely low humidity, and warm temperatures we see in the fall season in CA and you get a big fire.
Interestingly, controlled burning was a common tradition in native american cultures. Maybe we should've listened to the people who lived here for potentially thousands of years instead of oppressing the fk out of them to near genocide levels. But I digress.
The US Forest Service does do prescribed burns and the Smokey Bear thing is about people starting accidental fires — Smoky isn’t about opposition to prescribed burns.
Bad policy is a contributing factor to these fires. The Paradise Fire’s destruction and human toll was exacerbated because Paradise implemented “road diets” — squeezing critical evacuation and first responder routes. [1][2]
It’s a problem of policy being made by people that have no concept of unintended consequences: environmentalists opposing proper forest management as well as creating public safety risks by narrowing roads, “calming” traffic without the common sense to realize that roads serve an extremely valuable public safety function and that many so-called environmentalists promote policies that feel good to the simple minded, but actually do more harm. [3] It’s like the people that oppose hunting — they think they are saving an animal but they end up causing greater damage to the ecosystem because hunted species such as deer end up overpopulating and starving to death. You see similar effects with wild mustangs. People fiercely oppose culling herds even when not doing so results in mass starvation and disease that ends up harming more horses. The idea of shooting a wild horse gets “environmentalists” up in arms because they can’t see the forest for the trees.
My point is that forest and environmental management should be done from a position of rationality, reflecting reality rather than some emotional response. Examples: laws against cutting trees over a certain diameter without a time consuming permit process from CalFire. Opposition to logging, even when done responsibly. Protecting of trees when a spotted owl is in the area — even if those trees and associated underbrush become extreme fire dangers. More owl habitat is lost from an uncontrolled fire than would have been lost by properly managing it in the first place. You rarely see this sort of bad policy in other states even though other states do have wildfires. Californians do care deeply about the land — we want to preserve the beauty and majesty of the landscape — but some of us have no idea what we are talking about — we respond to emotional pleas rather than with informed maturity. It’s like having environmental policy designed by well-meaning, but naïve teenagers.
California has the 5th highest rate per kWh[1]. (#2 excluding Alaske and Hawaii)
Just like our sales tax rates, income tax rates, our gas prices (due to taxes), sanitation costs, ad nauseum. We pay excruciatingly high rates for everything, and get mindbogglingly low returns for the money.
CA gov. and utilities are a textbook case in poor management and government out of control.
I'll caveat that I've never lived in California...But in New Jersey, we have the same sorts of expensive pain that you just described...but at least your weather is better, and you have a tad more elbow room. ;-)
Not to be insensitive, but for a South African this is a kind of LOL moment. We have had power outages mess with experiments and universities since at least the early 2000's. That of course and the arson that burned down auditoriums.
But in terms of experiments, if you want your MSc or PhD then you better make sure that you can go a couple of days without power to the grid.
If you are in a rural area, especially a previously "white" area as the pre-1994 NP government called it, then you need to have your own > 5000 L water tank. The town where I went to primary school had 9 days without water or electricity a year or two ago. Probably half of the town (it's a small town though) never have water between 9PM and the next morning since they simply turn of the pumps (whether there is electricity or not). The government officials also steal water to go sell for $1 per 100 L to areas where there is no running water.
You can think of it as a kind of osmosis between previously affluent areas and areas that had been mud huts. If you think of it that way it doesn't sound that bad, actually, but I think the net position is not the issue, the real issue to me is the corruption and apathy within the government and the rate of improvement. Africa should be ahead of China in economic growth if you look at the youth percentage and potentially massive workforce.
If you want to completely redesign an entire country's power grid and make it 100% solar/wind, then South Africa is the perfect candidate. That is if you could forget about politics for a moment.
Lol what are you on? This was one days notice on a week's notice. Due largely in part because they neglected to maintain their infrastructure over the years so they can shave off operating costs - for better quarterly reports.
You know what can provide 8 days of backup power? A generator (along with periodic trips to the gas station). Which you should already have if your power needs are this critical.
Any dime a dozen facility with any sort of need of continuous electricity has a backup generator. The point is not that they should have bought a new generator for this planned blackout. The point is they should have had it for decades already.
I wonder at what point California will just nationalize their grid/energy infrastructure. These “markets” always end up as some private monopoly with all the corresponding terrible incentives around upgrades and maintenance. If it was actually controlled by the state, it could be mandated to build out renewable-centered, decentralized systems as well.
Well, in order to nationalize their power they would first need to be a nation. To be honest I don't know why one giant utility for an entire state is a desirable thing. In most places I've lived in, each community runs their own utility company. I've had cooperatives, city owned utilities, and private utilities. The best experience by far was the co-op. Since the customers owned the utility, we would get the profits back as credits on our bills usually during the middle of summer when power usage is highest in Florida.
>in order to nationalize their power they would first need to be a nation
A nation is a social construct that California could certainly fit into. Hell, Jerry Brown called it one. Examples of nations: Palestine, the Kurdish nation, Tibet.
The legal construct is called statehood. States can bind themselves into larger legal entities by giving up some of their own rights. Examples include: the US federal government, the European Union, and the United Kingdom.
Sorry, no. A nation is a legal construct, and California is not a sovereign nation. If it's a social construct then I'm founding my own nation tomorrow.
I don't know anything about Jerry Brown but he doesn't sound very bright.
A nation is a group of people with a common culture, identity, and usually language. A state is a legal construct implying self-governance and monopoly of force.
"Nation" has become colloquially synonymous with "nation-state", which lead to your confusion. But the legal aspect is from the "state" part, not the "nation" part. There have been historical states (eg. the Holy Roman Empire, Hapsburg Austrian Empire, Ottoman empire) that were not nations, and there are present day nations (the grandparent mentioned some: Palestine, Kurdistan, Tibet, as well as Basques, Zulus, and others) that are not states. Some present day nation-states (eg. Germany & Italy) were nations since medieval times but did not achieve unified statehood until quite recently (1871). The U.S. started as a state (technically, 13 states and one federated government) but did not really become a nation until after the Civil War.
If you can convince other people to identify with you, you absolutely can found your own nation. It won't be a state, though, so while might have your own customs good luck on having your own laws.
The US was most definitely a nation before the Civil War. And the Civil War was waged to prevent people that identified with each other from founding their own nation.
>A nation is a stable community of people, formed on the basis of a common language, territory, history, ethnicity, or psychological make-up manifested in a common culture.
California is part of the United States. The nation is United States. Sorry, I know Texans and Californians both think they're big enough to be exceptions, but they're still just states.
The stable community is the USA. We share highways, military resources, language, government, traditions, laws, etc. This is a ridiculous argument.
I highly disagree with this sentiment. Very few of the US institutions are well designed, and I suspect that there will be a decline of American hegemony worldwide.
People can, and frequently do, make distinctions between nations and states. Hence the term "nation-state" to refer to those nations which are also states. If there wasn't a distinction, the combined term wouldn't be necessary.
PG&E is big, but it doesn't cover the entire state. For example, much of San Diego is served by SDG&E, much of LA is served by SoCal Edison, and several larger LA satellites like Riverside (~400k pop) have their own city-managed power utilities.
They were nationalized, once, and were privatized by the electorate something like 20 years ago. I remember people debating it as a kid
edit - I'm looking this up and having trouble finding evidence of this outside of AB 1980 (proposition to break the state-sanctioned monopoly and allow competition among energy suppliers), which was from 1996
Interesting. Didn’t know that history. Kind of wonder what the arguments in favor of privatization were given that doing so doesn’t seem to have resulted in any meaningful upgrades or better outcomes.
"Private companies are more efficient" is one of those things that's been said often enough that it becomes a "truth".
This obviously doesn't pan out when you've got a monopoly or captive audience. Around me we have a choice of energy retailers, so we get to deal with the "efficiency" of several competing bureaucracies instead of the "bloated" government one.
Politicians love it because it abstracts them from any accountability to the people, even our left wing parties are pro-privatization these days.
Actually, India has public utilities and most of the common cause for loss is Theft, at multiple levels.
A private company would eradicate it since its effecting the bottom line, but public companies may choose to ignore because the people stealing may be a voting bloc to curry favor.
It may not be the case in US, but you are trading one set of problems with another and that is the best case.
I think by theft GP meant "random citizens stealing infrastructural elements made from copper for scrap value" and stuff like this. A private entity ready to steal from people or itself still won't tolerate outsiders stealing from them.
A private monopoly on its absolute worst day would still operate more efficiently than a federal or state power agency. One still has a bottom line, the other just has meetings to look busy.
Not really. A dysfunctional private monopoly will go as bad as they can go without having people just abandon a product/service category entirely. Which, in case of captive audience or a necessity, can be really bad. Oh, and with falling quality prices can still go up - it's a monopoly, after all, it's not like customers can go elsewhere.
I'm curious, do you think taxpayers ought to be on the hook when a state-run electric company burns down a town, or should California invoke sovereign immunity when that happens?
We tried nationalizing in Europe, within a capitalist framework: it just doesn't work. The TL;DR of 70 years of that experiment is that the State (gov and legis) is an extremely bad steward of any business endeavor; whereas lack of competition basically means no innovation and rent situations (it becomes sort of a tax in disguise to just hike prices or keep them stable when they should decrease).
Most unregulated big markets devolve into some kind of cartel after concentration, though, so that's yet another dead end after a 50-year cycle generally (telecom shows it well, TV as well, construction too).
What seems to work, for now, is a sort of regulated hybrid framework:
- networks and transport (of elec, data, whatever) is opened to a few companies, which can only sell access to said infrastructure to other actors, not market it to end clients.
- classic "operators" thus rent infrastructure capacity to these major actors and sell it to customers.
This allows very small actors to emerge (like "virtual" ISPs and telcos, we now have very small energy companies all over Europe). It's a recent development but these new startups provide great customer service and modern end/site equipment (easy to monitor, plug to some IS, etc).
The infrastructure caretaker status is typically "slightly public", with a non-profit mandate to maintain the infrastructure and develop it, using influx of for-profit investments by the operators, some citizen oversight, special status by law, etc. Basically, we try to make sure it costs as little as it should (it doesn't always work, rails are in a very bad situation for instance, and the caretaker is so deep in debt it's a failure, but the oversight of that dates to 100% public times with government-appointed CEOs).
TL;DR / conclusion: find the right medium between "private/unregulated" and "public/monopoly". Both are disastrous, but there are sweet spots in-between. Note that I have no idea how to pull off such change in the US, elec or telco markets are simply too entrenched / congress too corrupted it seems. We have that problem too but currently we're circumventing most democratic systems (whether citizen opinion or lawmakers corruption) by invoking the EU: “we must do it, no choice, it's EU law now”. That works most of the time and is the reason why the EU is so hated yet so quintessential. I'm not sure the Federal level would carry as much authoritarian weight in US States.
A counter example would be Quebec's grid which was nationalized in the 40s. Hydro Quebec owns generation, transmission and distribution for the province and provides the lowest residential rates in North America while operating at a profit. Admittedly we have access to stupid amounts of hydro power and a population that's a lot smaller than California.
I think the right thing is to go back and forth between the two systems. Wait until people are fed up with one, then switch to another to clean up and optimise. Once that stops working well switch to the other model. Rinse and repeat.
Maybe these kinds of things will push us more towards decentralized power generation like solar and wind, and bigger lab-wide or campus-wide battery battery backups?
Most electric utilities in most states aren't regularly given insane contradictory legal directives that can only be reconciled by cutting power. This is not normal.
PG&E is the poster child for regulatory capture. They're only in this mess because they've dragged their feet for so long on making critical upgrades and fixes.
The way I hear it they were required to make the critical upgrades and forbidden from charging high enough rates to pay for it. If this is true, then the people imposing those two constraints have only themselves to blame for the blackouts.
Regulatory capture is a thing that happens and is a problem, but this seems to be a case of bad regulation, not regulatory capture.
The way I hear it they were required to make the critical upgrades and forbidden from charging high enough rates to pay for it. If this is true, then the people imposing those two constraints have only themselves to blame for the blackouts.
Bullshit. PG&E redirected around $100 million from their safety budget and spent it on corporate bonuses instead. While hopefully not part of their safety budget, PG&E spends about $400 million annually on stock buybacks. PG&E has plenty of money to prevent this bullshit but is more concerned with executive compensation.
I mean, for fucks sake, Caltrans had to threaten to shut down the Hwy 1 and Hwy 24 tunnels before PG&E thought about bringing in generators (not even leaving enough time to install them before the power was cut).
Regulatory capture is a thing that happens and is a problem, but this seems to be a case of bad regulation, not regulatory capture.
Anything above 0% is atypical for a company in the financial position that PG&E claimed to be in. Paying dividends is something that profitable companies do.
Common refrain: let’s upgrade the systems. Ok! We charge users more. No, can’t do that, only the wealthy should pay for the changes. ... Nothing gets done
No, it's going to lead to even more regulation which in turn will constrain what PG&E can do even further. At this point, it should just become a state owned and operated enterprise and drop the dressing of being private.
Once it does that, we'll come to appreciate their service as much as we appreciate the schedulekeeping of the MUNI.
PG&E has a government-granted monopoly to provide its utilities to its coverage region. That makes it indistinguishable from a state-owner enterprise, except that there are probably even more people reducing efficiency by extracting value for themselves.
Extraction of value means increased efficiency is incentivized. Obvious caveats about cutting corners with brush removal but also the shared responsibility of homes built in dangerous places apply.
Where would that incentive come from? Not from the threat of competitors, because no competitors are allowed to exist. There would perhaps be some slight incentive due to the slight elasticity of demand for electricity (i.e. people can probably reduce their energy usage in response to higher prices), but I imagine that elasticity is quite low indeed.
It’s more likely that the more significant incentives would be other things, like, oh, not maintaining equipment and eventually causing wildfires.
This is interesting because if this happened in other states or countries, i might disagree with you...but considering it happened in California - where much innovation happens - i wonder how many existing and start-up firms will look to innovate on decentralized power generation even if only just for their own itch to scratch. I'm hoping that firms in California get pissed off enough to really increase the push for more decentralized modes of power...so instead of having engineers figure out how to invade our privacy or click online ads more, perhaps they can work on decentralized, reliable, and yes cheaper power generation.
Meanwhile in Germany the average duration of blackouts in 2017 was around 20 minutes per customer, in quest where it was pretty high because of storms. The frequency was 0.28 in 2017, which means you can expect every three years an outage.
That's because consumers subsidize the big smelters and other industrial customers who don't pay the renewables-buildout tax (EEG Umlage). The generalmtax load on energy is immense.
All of these big research universities in CA are on major and active fault lines. My old lab at Cal sat mere hundreds of feet from the Hayward fault that runs through campus. This is a proactive power shutdown when you know it's coming. If they can't an event with advanced warning, what will happen to hundreds of millions of dollars of frozen samples and research instruments when the unexpected earthquake hits any of these universities, not only disrupting power, but also disrupting structures? Incredibly foolish not to have at least reliable power backups and containments irrespective of what PGE is doing.
We're going to spend far in excess of 700 billion dollars to maintain our global military empire this year alone (not counting legacy costs for veterans and countless other huge, related expenses) while millions of Americans sit in the dark without power because we outsource our control of our power grid and other critical infrastructure to for-profit corporations. Hopefully some of those people sitting in the dark will give that some thought while the lights are out.
This isn't the best sourced article, the sources here are all random people complaining on twitter. It would be much more interesting to hear from an informed spokesperson from UC Berkeley, with actual data on the percentage of STEM research buildings without generators, etc. etc.
Even then, one of the sources did describe that they have an e-power plug in their lab, for the -80 freezer. That's pretty much how it's supposed to be. If they have additional important fridges, those should also be plugged into e-power.
I'm sure it's possible there are some labs at Berkeley that may be compromised in backup power, there are a lot of old buildings and infrastructure there. That still doesn't mean that that experience is common, just that the people impacted are complaining loudly. Most research labs may be totally fine, and just waiting for things to come back.
Curiously, a similar problem happened last week in the Institute of Natural Products and Agrobiology from the Consejo Superior de Investigaciones Científicas in Tenerife.
Temperature of ultrafrozen samples raised to -75 Celsius and years of hard work and a collection of DNA reference samples for agriculture were about to vanish
Berkeley, my post doc alma mater, really has fallen on hard times. I get that PGE caused this, but UCB should have reliable back up power for their labs. Full stop. You can see why they're fighting so hard in the CRISPR case. They're desperate to raise dough.
I think it is interesting to compare PG&E to how the electricity grid is owned and run in the UK. Many people here are saying they should be nationalised, well the UK did the opposite 30 years ago and it seems to be working fine.
In the UK there is National Grid Electricity Transmission plc, which as the name suggests is responsible for the transmission infrastructure across the country. They are a publicly traded company, and the parent company owns electricity & gas distribution in the UK, and some in the US. There are then smaller companies who are responsible for maintaining more local infrastructure (as I understand it's still owned by National Grid, but they are responsible for fixing and upgrading it).
As a consumer you can choose to buy electricity from a long list of companies, and easily switch between them. All that happens is a different company collects your meter reading and they send you a bill. A lot of these companies also own generation facilities (e.g. SSE, who are also publically traded) while some are just resellers (e.g. So Energy).
I'm in NY, and the gas & electric company is National Grid, and they do give you the option of choosing a supplier.
I've always thought that there must be hidden pitfalls of selecting a different supplier, since National Grid would have the ability and incentive to tilt the playing field.
There are plenty of states where that's basically the system that are a mess, if not as bad as in California. The devil is in the details of how the post-privitization market is regulated.
Portable generators are so inexpensive that I'd think an institution like this could just keep some in a shed or garage, and shlep cords through windows for extended outages. I bought a top-of-the-line tiny Honda Inverter generator so I could keep my wife's breast milk frozen in case of an extended outage, but even something half the cost would be good enough to run a refrigerator.
But, even then, if you plan for it, backup generators and Solar + Battery are very affordable!
I recently installed a Powerwall purely for backup. Because I have solar, it's slightly more expensive than a standby generator, but with no maintenance or fuel I suspect it'll be a wash within a few years. (Backup generators need expensive maintenance every year.)
Even Generac bought a Solar + Battery company because they know they can't compete, and their solution is nicer than my Powerwall! (The Generac solution includes a critical loads panel inside of it, unlike Powerwall where the critical loads panel is outside.)
I have no idea why this shutdown was initiated. It had been much warmer and drier earlier in the summer. It is actually quite pleasant and cold these days, with morning dew. And no winds. I have no idea why these conditions warrant power shutoff
The intensity of these blackouts is driven not just by weather, but by a major change in risk assessment by the utilities. PG&E, the utility that services most of northern California, was sued by a number of insurance entities after it was found that their equipment was the cause of several major wildfires over the last few years. Most notably the 2018 Camp Fire, which was the deadliest and most expensive wildfire in California history.
But most importantly, it's what didn't happen: the state of California did not bail out or grant PG&E immunity from these lawsuits. PG&E settled with the insurance entities and has since filed for bankruptcy protection, since the possible liability is larger than their market cap.
The blackouts are a way to minimize further risk and liability. But I would also guess that the intensity of blackouts is a not so subtle middle finger by the utilities to the state government for its resigned treatment of these issues.
> It had been much warmer and drier earlier in the summer.
Warmer yes, but dryer definitely not. Earlier in the summer there is still moisture left over from the previous rainy season. By mid fall it is much dryer, and the risk is increased by offshore winds coming from inland.
It was this time last year that an entire town got wiped off the California map and 80+ people died because of fires sparked by PG&E's horribly-maintained power lines. You think they want that to happen again?
I go to UC Berkeley. My midterms were postponed so I feel like even though I should be complaining, I really can't. PG&E basically gave me a week extra to study.
all the SREs on the thread are wondering why scientists don't automatically mail their complex samples to their friends across the country as a backup.
and the answer is.... individual PIs optimize for the easy case (that power stays on), not the disaster recovery scenario. There is no SRE thinking at the individual PI level.
Rather than answer several individual posts, I'll be lazy and do a larger post which involves a lot of simplification and hand waving. I apologize in advance for any errors, but this seemed like a good way to handle insomnia instead of my normal lurking. If something is unclear or you think wrong, let me know.
Power Companies
circa 1900 there were lots of power companies, I've seen pictures of urban areas where there were multiple circuits run on poles, how many companies tried to serve a certain area, I do not know. Through lobbying investor owned utilities (IOU), or in most cases a single IOU gained the rights to serve an area exclusive. IOUs concentrated, for the most part, on serving dense load concentrations. As such the US Govt implemented the Rural Electrification Act, so low density areas could be served. The 2 types of electric utilities are public power (municipal, REA, RUS, PPD, UD, etc) and for profit (IOU's typically). In most cases, the IOUs give something to the government in exchange for the IOU providing service exclusively, it might be a tax or it might be free street lights. Municipals tend to subsidize the local government in some way, either through returned dollars or free power (street lights, buildings, traffic lights). IOUs typically have a defined rate of return and are supervised by a governing entity of some sort.
Vegetation Management
The recent SERC compliance meeting had a good presentation on vegetation management from the utility perspective and another on enforcement trends. In my opinion a lot of the issues in the W US is the result of the policy 'no fire is good', the sand pile game I think illustrates the issue where the longer sand keeps from falling results in a larger collapses (see Yellowstone fire). Since the late 1980's the issues with not burning has been known, I had an ecology class where if I recall correctly that was discussed for a couple of days. Tree trimming and clearing out undergrowth is done on a regular basis when the utility has an easement, but especially in urban areas folks tend to plant trees too close to power lines or even worse encroach on the easement with buildings. Most utilities patrol transmission lines at least once a year if not twice or monthly, sometimes this is aerial and other times it is feet on the ground walking the line. As an aside, the NESC governs clearance of electric lines to stuff and how stuff should be built; the RUS has publications on line design if you want to read about it.
Distribute Generation
The electric grid in the US is divided into 3 areas, Eastern Interconnect, Texas and Western Interconnect; they all function essentially the same. If I have a generator connected to the grid, it has to synchronize to the grid before closing the breaker. If it is done correctly then there is very little mechanical stress on the generator, if done incorrectly then there is a large amount of mechanical stress on the generator. One mis-operation I know about involved the A and B phases being swapped during a re-wind, when the generator was closed in at commissioning it had a large bang/clunk and the breaker opened immediately. The generator then had to be examined, i.e. taken back apart, to figure out what went wrong and if it could be put back into service.
If I have a generator partially supplying a facility (this can save a lot of money for an entity) and a fault happens on the grid then my goal is to protect the generator, so the generator will either shut off or island the facility while shedding load above the generator's capacity. This happens very quickly. One instance I know of, the urban area was supplied by transmission (aka remote generation), a 30 MW generator was the closest source, the utility had a fault because equipment misoperated and the generator was suddenly trying to supply all the power to that fault such that the generator protection operated and islanded the facility. It was no issue to close the grid interconnect back in (once it was ensured it was safe to do so) but the facility had to shed load to keep the generator running without causing electrical issues to load and damaging the generator.
Once a facility is islanded and running on its own generation the phase angle is a don't care until it is time to synchronize back to the grid. As long as the facility can shed load to maintain frequency (there is a NERC standard on Under Frequency Load Shed if you want to read about it) and not ruin equipment by having a power quality issue. During dynamic studies for system stability, it can be observed that a generator will diverge from the system frequency phase angle but not trip off because it is isolated from the grid which requires verification that isolation is happening and the protection scheme will indeed work that way on the actual system.
My observation is that most facilities, data centers and other processing facilities (refineries) tend to be the exception, concentrate on first costs when designing their electric infrastructure. It is possible to design a resilient system but it has a cost and it will not be utilized 100% until something goes wrong. And if you are doing research then that can be an issue as you may lose a large amount of data due to the power going out or possibly being sensitive to transients on the system, e.g. a switching operation on the transmission system affects the end user equipment. Even if you have redundant systems (and/or power supplies) it is possible to have single point of failures on your system. As well if you have enough local generation to supply your load, it may be more economical to not run 100% of your generation as the market price for electricity is cheaper than your cost of production (and there are folks who don't like idle assets, not realizing the greater benefit is not using it or only having to use it infrequently).
One other aspect of distributed generation is the automatic separation of the DG when loss of voltage is detected on the grid side. Utilities do not want voltage on their system if they have an outage due to worker safety (and other reasons). Utility crews in hurricane areas will typically investigate if they hear a generator running when the power is out to an area to ensure it is not back feeding the distribution line. As a reminder keep your feet together if you are near a downed power line and hop away, or even better don't go near downed power lines.
Yes (IMHO). PG&E was found responsible for the biggest/worst fire(s) in recent history, and has since declared bankruptcy. They're giving us Californians a big middle finger. I mean really, suppose a fire were to start on the Berkeley campus -- that would not trigger a forest fire miles away.
I wouldn't say it's as emotional as a middle finger. Businesses manage risk in different ways, and one of the types of risk is legal risk. Based on new information that this risk is MUCH larger than they had previously accounted for, they are now making adjustments for it.
They can't afford another lawsuit like that one. They couldn't afford the one that already happened. So they are taking steps to prevent it.
It might be a good idea to study the physical geography of the Bay Area and read up on the history of the Oakland Hills fire. The Berkeley Campus is much larger than most people are aware of and extends up into the surrounding hills.
It’s not because of the lawsuit. Their grid really is dangerously under-maintained and prone to bursting into flames. Turning it off is the rational short-term move.
If you have work that can be destroyed by a power outage you should plan accordingly. Power outages are a normal thing and can happen at any time, especially in areas with earthquake and fire risks. Not preparing is foolish.
For academic scientists, it's not always up to us. Just going out and renting a generator and hooking that up to our instrumentation ourselves may be technically possible (plus or minus various building code and safety issues), but by that time the damage is probably already done, so backup power really needs to be formally installed. We can beg the facilities and maintenance dept to install proper backup power, but unless we managed to get that written into our startup budget at the time of hiring, the money is probably not going to appear.
Then you should not do it. It is a calculated risk and you have not be able to afford it, not doing it or accept the consequence. Frankly cannot have 99.9999 power in any case. Too expensive in the power company side. Average and the special issue.
They don't need 99.9999% power uptime, it is common to have UPS backups for critical equipment to cover brownouts and even day-long outages. This has been going on for 3 days and might last 5. That's not even 90% / one-nine uptime on a monthly basis.
>you have [to] be able to afford it, not [do] it or accept the consequence
What 4th option are you criticising? Unless you're saying that if they don't immediately throw up their hands, try to do nothing, and go home when the power goes out, they're not "accepting the consequences"?
> Power outages are a normal thing and can happen at any time
I cannot recall a single power outage in my home city that lasted longer than ~three hours. And even those "couple hours" outages are extremely rare, much less than once a year I would wager.
According to the statistics, the expected unavailability in my country is around 0.002 - 0.003 %. That's hardly something I would describe as a "normal thing" that "can happen at any time". That's a really rare thing.
The "SAIDI" value here is under 15 minutes. Meanwhile the median SAIDI of the US is ~1.5 hours. That still doesn't sound like a common thing.
I lived in a very nice north shore suburb of Chicago for about 10 years, and we had one power outage that lasted four days. If I had critical stuff going on at home, like some key medical equipment, or other such process, I would make an investment in finding out how to cover for a multi-day outage.
So, be corrupt? There’s no other way to describe a deal that a large company can make with the electricity company to keep its electricity running when the electricity company shuts off everyone else’s power.
How’s that corrupt? If they have an agreement there are likely penalties if they can’t meet the agreement. I worked for a company that was losing a million dollars a day because they couldn’t provide a critical service to a manufacturer.
If it's "corruption" to pay more for greater reliability, it's "corruption" to pay more for a large sandwich than a small one. There's nothing wrong with the quality of good or service varying according to how much you pay.
Presumably, if you are paying more, then part of that increased amount you are paying is to make certain that the portion of the grid needed to keep your factory going is properly maintained. Unlike the rest of the grid. Which would apparently be too expensive (i.e. everyone on the grid would need to pay the extra fees for increased uptime).
:::erm::: The State Gov and PG&E. Do you think there isn't corruption right there? The more regulation, the more opportunity for extraction/corruption.
Who's doing the inspections, who gets contracted to clear brush, who gets power, who doesn't, etc.
It's kinda like MUNI, just not quite there, but I don't doubt Sacramento can get them to EDELCA.
Plus it's... California. UC Berkeley is literally a 44 mile drive (reported as 1 hr 19 min right now) from Tesla's HQ. Generators do present some retrofit issues, but now that I read this I'm actually moderately surprised they haven't long since investigated at least a few Powerwalls for various labs. UC Berkeley has a $4.5+ billion endowment IIRC, and on the scale of lab equipment, grants, and how much value would be lost if even a single experiment or sample set was in fact destroyed it does seem like a weird oversight. PG&E's woes are hardly breaking news either, nor are fire seasons even ignoring any other disasters.
I mean, I'm used to IT in general, and security in particular, being treated as a cost hole and given as skimpy as budget as possible and then some until something breaks. But at least that's more intangible, and I guess from a cold hold MBA point of view it's not as if most places are really punished that hard for security breaches unfortunately. But power loss and having valuable work/material lost every year seems like it should be more tangible.
They do plan accordingly, but not for this kind of extended outage. Labs with critical equipment always have backup UPS or two (or 3!), but only enough to last for hours or possibly a day. The UC-Berkeley campus is going on day 3 without power, and PG&E has said it may be 5 days.
They're silently making sales around the country. I've seen their installations at a few places in the Bay Area. My guess, tacitly acknowledged by a friend I bumped into several years ago who was a sale rep, is that because their technology doesn't have strong patent protection (just a productization of long-existing tech), and the margins slim to none, they're careful to avoid inviting competition, and presumably can't afford to burn much money on flashy campaigns, anyhow. Hopefully the outages will give them a bump as it seems like precisely the sort of no-nonsense solution to on-site power generation needed by light industrial customers, such as labs and smallish data centers.
It’s really difficult to beat diesel for many reasons, sure solar + batteries is with enough for though. But what if you need to augment beacause some were damaged in a weather event, IR dropped us a secondary genset next day after a tornado (mind you precontracted). Other fuel options exist but you can’t rely on piped natural gas in a highly active seismic zone (as they shut off entire areas after an earthquake to limit leaks). Diesel is stable for long periods, if it’s inundated with washer can be mechanically separated, then you add on the safety aspect of it being less ‘energetic’ than gasoline.
Not saying there aren’t other options, just the one most businesses have opted for, and has the best ‘vendor’ support (plenty of fuel companies to deliver, rental agencies to arrange the delivery, and companies that Veratis that setup the building side needs)
I’m merely talking about situations where everything is fine except for the occasional brownout or blackout.
We are trying to move towards greener electricity. Wouldn’t it make sense for parts of California, for example, to generate some of that electricity locally?
Baffling that they couldn't have a diesel generator handy. One can rent an inverter, like for portable lighting units or a Lincoln welder, pretty readily and for relatively inconsequential sums in most cases.
They are easy to acquire, as long as everyone else doesn’t need one at the same time. Heck even just securing a diesel delivery during a widespread event can be impossible, which is why you need some onsite for multiple days (if possible).
Sadly you need to plan for these well in advance and be willing to sink significant resources.
This is probably the reality now. Unless we're going to make the investments to actually maintain and upgrade the grid and infrastructure, preparedness and whatever amount of self-sufficiency that can be achieved is going to be necessary. It is a horrible cliche, but failing to prepare is preparing to fail.
Possibly this is a case of California climate biting them. A lot of the big research universities in the Northeast have steam plants, originally for heat, but often with cogen facilities as well. Usually not much, a meg or two, but it is something.
I suspect in most cases, those prices assume that demand is pretty low. When you have a regional power outage, it seems likely that limited supply will raise the price of these backup power.
The issue at hand is dry brush and trees being blown across live wires and sparking wild fires. The blackouts are targeted at the areas where this is most likely to occur. Namely, not heavy urban areas.
The problem is lack of investment in the infrastructure. And that damage to customers or economies is not a concern compared to lawsuits damaging the bottom-line.
In particular, if you only take into account the revenue made on a given day vs the likelihood of a lawsuit and nowhere is economic damage accounted for, then you will turn the power off a lot more often and a lot more widely than you should.
Also, since SoCal Edison is geographically adjacent to the DWP but Edison is investor owned while DWP is municipal owned, I believe you are adding support to my point.
Exactly, the question we should all be asking is how it got to this point. I reckon you'll find a lot to blame with both the utilities and state government, and the existing incentives for both.
However, to your point about LADWP, I think you're making an overeager classification. It's not a matter of being "geographic adjacent", but being liable for the infrastructure that would start wildfires in at-risk regions. Take a look at the difference in coverage areas of Californias utilities: https://ww2.energy.ca.gov/maps/serviceareas/Electric_Service...
edit: it is tangential to this issue, but LADWP does not have a very glamorous history, either.
Exactly, the question we should all be asking is how it got to this point. I reckon you'll find a lot to blame with both the utilities and state government, and the existing incentives for both.
No this is solely PG&E's doing. When you write the regulations, you own the blame. PG&E claims it's too expensive to underground its transmission lines (which would prevent fires like these) at a bit over a million per mile. Meanwhile they took blew their safety budget on executive bonuses. You want to tell me that undergrounding around 100 miles of lines isn't going to reduce the risk of fires? How about undergrounding 500 miles? Because in addition to the safety budget, PG&E also spends around $400 million annually on stock buybacks.
I don't particularly care if PG&E remains a private company, but there need to be consequences like jail time for actions that go far beyond mere negligence.
You're arguing against a straw man. I agree with you: PG&E has acted terribly. The attempted executive bonuses (currently being denied by a court) and stock buybacks ($250MM in F'18, $395MM in F'17, and $968MM in F'16) are especially egregious. The company should no longer exist.
But PG&E are not the only offenders here. You seem to be arguing that there's regulatory capture (correct me if I'm wrong), but the Public Utilities Commission still exists, and they have the authority to regulate PG&E. If the commission is neglecting its duties, then it should be restructured as well. A president of the commission has been forced out before, and it should probably be done again. If they neglected such an obvious case of greed and mismanagement because of bribery (sorry, lobbying), that should be a criminal case, too.
Public officials can throw PG&E under the bus all they want, but if systematic changes aren't made, we'll be doing this song and dance again in a few years with a different utility.
The dangerous combination that warrants blackouts is high winds, dry brush, and exposed live wires.
While this combination is more likely in a desert, it's not true of all. For instance, the Salt River Project that oversees electricity transmission in Arizona switched to building underground lines in the 1970s. https://www.azcentral.com/story/news/local/asked-answered/20...
Surely, PG&E will learn their lesson during this outage and send crews across the state to trim brush along the transmission lines? Strengthen the towers?
Uncharacteristic edit: Why the downvotes? It must be because you took the comment seriously and think the sentiment grossly naive, or heard the skepticism and thought it unwarranted?
One day some years ago, the university administration thought, "why are we wasting money keeping power on, running lights and aircon and God knows what, when the buildings are vacant?". They soon turned that thought into action, and when a long holiday break came, they just shut off the power to the buildings as soon as everyone left for the day. About a week later, the PhD I mentioned and his co-workers came back to the lab, saw some droplets on the glass, and almost had a heart attack.
Turns out, after a week of being without aircon to dry and warm it, the air was just about to cross the dew point threshold, and if it did, it would kill the optics on a million-dollar research laser they recently got installed.
I hear that since then, they're a little bit less pound-foolish.