Balancing the needs of many users is hard. The gift of having a quirky setup is it improves inclusivity by default. In addition to using mainstream browsers with and without screen readers, I browse through other means: I often read by piping Readability output (using rdrview) through a textual browser, using non-mainstream browser engines (e.g. NetSurf), using a textual feed reader with a bespoke markup renderer (Newsboat), and using the Tor Browser’s “safest” mode.
Simply getting my content to work for me will automatically make it work for a wide range of audiences. Of course, this alone isn’t enough; there are others still excluded that I need to account for.
Web devs: make your setup weird. It encourages you to make your site robust, less dependent on a narrow range of implementation quirks or poorly-followed standards.