Avraham Schwarz is the son of Izador Schwarz and Leonia Grun Schwarz. Izador was born in Sered on January 5, 1909 and Leonia was born on September 30, 1907 in Tecin. Izador was the son of Moshe Aaron Schwarz and Ottilie (nee Weiss) and was one of ten siblings. Leonia was the daughter of Abraham and Mia Gruen. She also had many siblings. Izador's uncle, Heinrich Schwarz was the leader of the Slovak Orthodox community. This brought him to the attention of Dieter Wisliceny, Eichmann's advisor, who appointed him the head of the first Judenrat in Slovakia. This enabled Izador to work there as a secretary. Though some Zionists initially opposed the appointment of Schwarz they soon became convinced that he was doing his best to protect the community. In fact, the following year, the government deposed and arrested him for non-cooperation. Gisi Fleishmann, the former head of WIZO (the women's Zionist organization) and the Central Jewish Relief Committee also worked under the auspices of the official community. She founded a resistance group known simply as the Working Group (Pracovne Skupina) to save as many Jews as possible and hired Leonia to work as her secretary. After the Slovak uprising and disbandment of the Working Group, Izador and Leonia survived in hiding until liberation. After the war, Izador worked for Rabbi Frieder before immigrating to the United States in 1948. They went on to have three children. Naomi was born in Slovakia 1947; Avraham and Jay were born in the United States in 1949) and 1953, respectively.
Most of the rest of their families perished in the Holocaust. Izador's parents Moshe Aharon and Ottilie Weiss perished in Auschwitz as did several of Izador's siblings. His sisters Berta (born in 1898), Rozi (born 1896) and Sari (born 1900) perished in Auschwitz in 1942. His brother Aladar (b. 1911) perished in Auschwitz in 1944, and his brother Jeno (b. 1902) perished in Kosice 1944. His brother Beno (b. 1906) perished in Sachsenhausen in 1945. Leonia's father Abraham Grun (b. 1863) perished Auschwitz 1942 together with her mother Mina Gruen who was born in Oswiecim in 1869. Leonia's sister Regina (b. 1893) perished in Auschwitz 1942 and her brother Dawid Grun (b. 1902) perished in Auschwitz in 1943.