Geometry-charge responses in topological phases of matter