Responding to Outside Pressure

When looking at the progression of the halachic world of the last 1400 years since the closing of the Talmud, one can see roughly fourth eras. The first is the post-Talmud period where the Geonim and the Rishonim developed, expanded and organized the law in resopnse to changing and developing circumstances as well as the need to clarify rulings from the Talmud that might not have fully accounted for new situations. The second era is the rise of the formal Codes, start with the Rambam and the Rosh and culminating in the publication of the Shulchan Aruch. The third is the early Acharonim period where commentators fleshed out the Shulchan Aruch, again in response to new and changing circumstances.
The fourth, however, is a period of near-ossification of the halachic process. As opposed to previous eras, this one developed in response to a threat to the authority of the Torah from within the Jewish community, specifically the rise of Reform and other Jewish movements that claimed that one could be a good Jew without abiding by the observance of Jewish law. The defining statement of this era was, and remains, the Chasam Sofer's strident declaration: anything new is forbidden by the Torah.
Reviewing the previous eras of halachic development shows just how concerning this attitude has become. Before the Chasam Sofer's declaration, halacha was a living entity, growing and adapting over time, doing its best to fit the needs of the Jewish people while keep that people faithful to the Torah and service of God. After the declaration, change becomes suspect and, in most cases, forbidden. Any new situation becomes an issur, with the reason why the only thing to be determined.
now, this is a blanket stataement and not completely applicable nowadays. After all, in contrast to the far more consistent Amish and Mennonites, Satmar chasidim drive in cars, have electricity and running water in their homes and use toilet paper just like the rest of us despite their opposition to "innovation". Even in deepest, darket Mesach Shearim, time has not been frozen. There's just a refusal to admit they can't freeze it.
But the root cause seems to always come back to the same thing - the rise of Reform. The reaction to a movement which championed innovation pretty much for the sake of innovation lead to a strident reaction that all change was forbidden. This results in a hobbling of the halachic process that continues to this day.
And that's a teerrible shame. For one thing, it turns a complex subject that God meant for us to study, understand and develop into a simplified tool for forbidding everything it can. And for another, it hampers Torah Judaism's ability to encounter and thrive within the modern world. Consider a fascinating historical note from this article:

The second and more recent cause of extremism, says Sperber, is the central role that the large yeshivas and their heads began to assume in the Ashkenazi (European) halakhic world: "In the past it was common for the pesika to be issued in each community by the local rabbi. The rabbi was familiar with the nature of the community, its ability to observe various stringencies and the needs of the people, and therefore the decisions suited the community. The moment that the yeshiva heads became the main poskim the rulings became 'academic,' issued from an ivory tower, unconnected to the actual situation of the public, and in any case also tending to be stringent." Incidentally, he points out, because the two phenomena - Reform Judaism and the large yeshivas - were not found in Sephardic Judaism (in southern Europe, North Africa and the Middle East), the phenomenon of halakhic extremism was avoided there.

The Sephardic approach to Torah and halachic development has remained far more normative because of the lack of reaction to outside, heretical approaches. Yet because of the deeply-ingraned reactionism within the Ashkenazic world, this far more genuine and practical approach reamins disdained by those outside it.

Consider further the idea of accessing halachic works which might not have been known to the authorities of the major Codes:

As a basic example of that he mentions that Rabbi Yosef Caro, the compiler of the Shulkhan Arukh, which from the time it was written in the 16th century up to the present, has been considered the central book of halakha in the Jewish world, made his halakhic decisions by following the majority opinion in three previous books of halakha with which he was familiar: those of Maimonides, the Rif (Rabbi Yitzhak Alfasi) and the Rosh (Rabbi Asher ben Yehiel). "But in recent generations," says Sperber, "many halakhic sources that were totally unfamiliar to the author of the Shulkhan Arukh have been discovered, for example extensive halakhic literature from Provence. This literature could change the entire balance of majority and minority opinion in halakha, but the poskim will not allow any expression of that. The Hazon Ish [a central posek who lived in Israel in the 20th century -Y.S.] even said there is no need to take this literature into consideration, because apparently it was determined from above that this literature would not be discovered before the halakhic tradition was consolidated. Incidentally, Shas spiritual leader Rabbi Ovadia Yosef is actually willing to rely on the new literature, and in several places he rules contrary to the Shulkhan Arukh, in accordance with new sources that have been discovered."

If the study of Torah is a study for truth, and the practice of Jewish law the quest for a true spiritual connection with the Ribono Shel Olam, why would it make sense to ignore any geniune Torah knowledge simply because it didn't make it into the "official books"? Does one believe that if the Rav Yosef Karo had discovered these texts while writing the Shulchan Aruch that he would not have taken those opinions into account?

May God show us a true way towards His service and may we learn within and develop the halacha without insecurity and worry that our desire to treasure and build on His Torah will be seen by others as an admission of the rightness of their wrongess. We owe God our best effort to determine His Truth. Nothing should dissuade us from that.