You can remove silicone caulk from brick by heating it up and scraping it away. This won't remove it all you will need a wire brush to remove the rest.
First use a razorblade or sharp knife to remove the larger portion of the silicone. Once that is done use a rubbing alcohol and a rag to dissolve the silicone and rub it off.
Using a heat gun, warm the caulk and scrape off with putty knife or like instrument. Use steel bristle brush for remaining debris, with care not to damage the brick.
You can scrape the silicone caulking off the back of your bricks by using something to scrap it off. Putty knives work good, if you don't have one of those you could probably use a box cutter.